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FOREWORD 


In  March  1995,  the  Deputy  Assistant  Secretary  of  the  Army  (Civilian  Personnel  and  EEO 
Policy)  and  the  Director  of  Civilian  Personnel,  Office  of  the  Assistant  Secretary  of  the  Army 
(Manpower  &  Reserve  Affairs)  authorized  work  to  develop  an  Artificial  Intelligence 
(AI)/Robotics  civilian  specialty  program.  Under  sponsorship  of  the  Director  of  Information 
Systems  for  Command,  Control,  Communication,  and  Computers  (DISC4),  the  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  contracted  with  the  Human  Resources 
Research  Organization  to  initiate  this  effort. 

This  project  encompassed  three  goals:  (1)  identification  of  the  population  of  Army 
civilians  working  in  AI/Robotics;  (2)  identification  of  the  competencies  required  to  proficiently 
perform  AI/Robotics  tasks;  (3)  determination  of  training  and  experience  required  for  certification 
(i.e.,  the  certification  standards).  AI/Robotics  certificants  will  have  an  additional  skill  identifier 
of  AI/Robotics  added  to  their  personnel  records. 

The  results  of  this  effort  were  briefed  by  DISC4  to  a  wide  variety  of  interested  Army 
parties  during  the  2nd  and  3rd  quarters  of  FY 1997. 


ZITA  M.  SIMUTIS 
Technical  Director 
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DEVELOPMENT  OF  AN  ARMY  CIVILIAN  ARTIFICIAL  INTELLIGENCE  (Al) 
SPECIALTY 

EXECUTIVE  SUMMARY  _ 


Research  Requirement: 

As  the  Army’s  proponent  for  artificial  intelligence  and  robotics,  the  Director  of  Information 
Systems  for  Command,  Control,  Communications,  and  Computers  (DISC4)  is  developing  and 
implementing  a  process  for  awarding  specialty  certification  to  Army  civilians  who,  regardless  of 
career  program,  have  demonstrated  proficiency  in  competencies  related  to  artificial  intelligence 
and  robotics.  The  purpose  of  this  project  was  to  (a)  identify  the  population  of  Army  civilians 
working  in  AI/Robotics,  (b)  identify  the  competencies  required  to  proficiently  perform 
AI/Robotics  tasks,  and  (c)  determine  the  training  and  experience  that  will  be  required  for 
certification  (i.e.,  the  certification  standards).  Note  that,  for  the  purpose  of  this  report,  the  term 
“AI”  encompasses  both  AI  and  robotics. 

Procedure: 

The  basic  methodology  of  the  project  was  to  conduct  a  job  analysis  to  identify  AI/Robotics- 
related  job  tasks  performed  by  Army  civilians  and  the  knowledge  areas  (i.e.,  competencies)  that 
are  necessary  to  successfully  perform  them.  Competencies  identified  as  the  most  important  to 
successful  job  performance  across  different  types  of  Army  civilian  jobs  form  the  basis  for  the 
certification  standards. 

Four  one-on-one  interviews  and  two  workshops,  involving  a  total  of  13  Subject  Matter 
Experts  (SMEs)  and  two  officers  from  the  Army  Artificial  Intelligence  Center,  were  used  to 
develop  comprehensive  lists  of  AI/Robotics-related  job  tasks  and  associated  competencies.  These 
lists  were  incorporated  into  a  survey  that  was  sent  to  all  Army  civilians  involved  in  the 
AI/Robotics  work  that  could  be  identified.  Survey  respondents  rated  the  importance  of  each  task 
and  competency  to  their  individual  jobs. 

A  “data  call”  survey  was  used  to  help  identify  the  population  of  individuals  who  should 
receive  the  job  analysis  survey.  Surveys  were  sent  to  these  405  persons.  Although  240 
respondents  returned  the  survey  (for  a  response  rate  of  59%),  a  relatively  large  proportion  of  the 
respondents  reported  that  AI/Robotics  responsibilities  accounted  for  none  (68  people)  or  only  1 
to  5%  (50  people)  of  their  job  responsibilities.  Therefore,  the  analyses  to  help  identify  the 
competencies  to  incorporate  into  the  certification  process  were  based  on  only  the  121  individuals 
whose  AI/Robotics  responsibilities  comprised  at  least  5%  of  their  jobs. 


vii 


Findings: 


The  importance  ratings  from  the  job  analysis  survey  were  used  to  identify  competencies  that 
are  very  important  across  a  broad  range  of  jobs  (e.g.,  across  different  major  commands).  These 
were  identified  as  core  competencies.  Competencies  that  were  particularly  important  for 
respondents  working  in  particular  environments  were  identified  as  supplemental  competencies.  A 
workshop  with  six  SMEs  was  held  to  determine  the  final  core  and  supplemental  competencies  and 
decide  what  experience  and  course  work  would  be  required  for  certification  candidates  to 
demonstrate  proficiency  in  the  applicable  areas.  Their  initial  recommendations  were  compiled  into 
a  package  for  final  review. 

The  task  force  and  one  additional  SME  made  suggestions  for  minor  changes  to  the 
certification  standards.  The  task  force  met  on  April  7,  1997  to  discuss  the  changes.  The  meeting 
concluded  with  all  of  the  task  force  members  agreeing  to  the  revised  certification  standards. 

Utilization  of  Findings: 

To  be  certified,  applicants  will  need  to  demonstrate  that  they  have  had  sufficient  amounts  of 
education  or  experience  or  combinations  of  the  two  for  each  of  the  6  “core  competencies"  and  for 
three  of  the  18  “supplemental  competencies.” 

The  certification  standards  will  be  used  to  guide  a  review  panel  in  deciding  whether  or  not  to 
certify  individuals  in  the  new  Army  civilian  specialty  of  AI/Robotics.  The  AI/Robotics  course 
database  developed  for  this  project  will  also  aid  in  professional  development  planning  for 
AI/Robotics  professionals.  The  job  analysis  results  can  be  used  to  facilitate  staffing  and  training. 
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DEVELOPMENT  OF  AN  ARMY  CIVILIAN 
ARTIFICIAL  INTELLIGENCE  (Al)  SPECIALTY 

INTRODUCTION 


Background 

This  project  was  sponsored  by  the  Director  of  Information  Systems  for  Command,  Control, 
Communications,  and  Computers  (DISC4).  Its  purpose  was  to  develop  standards  for  the 
voluntary  certification  of  US  Army  civilians  working  in  Artificial  Intelligence  (AI).  The 
certification  program  is  intended  to  (a)  create  a  coherent  career  path  for  AI/Robotics  specialists 
and  (b)  fully  utilize  the  skills,  knowledges,  and  abilities  of  AI/Robotics  personnel  by  creating  a 
comprehensive  program  that  focuses  on  the  management,  development,  utilization,  recruitment, 
and  retention  of  AI/Robotics  specialists. 

Approval  for  the  AI/Robotics  civilian  specialty  program  was  given  in  October,  1994.  In 
March,  1995,  the  Deputy  Assistant  Secretary  of  the  Army  (Civilian  Personnel  and  EEO  Policy)  / 
Director  of  Civilian  Personnel,  Office  of  the  Assistant  Secretary  of  the  Army  (Manpower  & 
Reserve  Affairs)  authorized  the  work  required  to  support  this  effort. 

AI/Robotics  certificants  will  have  an  additional  skill  identifier  of  AI/Robotics  added  to  their 
personnel  records.  A  certificant  in  AI/Robotics  should  be  able  to  move  from  one  AI/Robotics 
position  to  another  with  little  additional  training  in  AI/Robotics. 

Summary  of  the  Project  Approach 

The  project  involved  three  phases:  (a)  identify  the  population  of  Army  civilians  working  in 
AI/Robotics,  (b)  identify  the  competencies  required  to  proficiently  perform  AI/Robotics  tasks, 
and  (c)  determine  the  training  and  experience  that  will  be  required  for  certification  (i.e.,  the 
certification  standards).  The  second  phase  was  accomplished  via  a  job  analysis.  The  job  analysis 
information  was  obtained  primarily  via  meetings  with  groups  of  AI/Robotics  Army  civilians  and  a 
job  analysis  survey  sent  to  the  entire  known  population  of  AI/Robotics  Army  civilians. 

The  determination  of  the  certification  standards  was  the  last  step  in  the  following  sequence: 

•  set  up  a  task  force 

•  obtain  information  about  tasks  and  competencies  via  1995  Informal  Survey 

•  interview  incumbents  to  refine  lists  of  tasks  and  competencies 

•  conduct  workshops  to  refine  lists  of  tasks  and  competencies 

•  locate  job  analysis  survey  population  via  1996  Data  Call 

•  develop  and  distribute  job  analysis  survey 

•  analyze  survey  responses  to  determine  important  tasks  and  competencies 

•  conduct  workshops  to  determine  certification  standards 

These  components  of  the  project  are  briefly  described  below. 
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Set  up  Task  Force 

A  task  force  of  four  Army  civilians  working  in  AI/Robotics  was  set  up  by  the  sponsor  to 
help  plan  and  execute  the  project.  Several  task  force  meetings  were  held  during  the  project.  In 
addition,  the  task  force  members  reviewed  materials  and  participated  in  workshops.  The 
characteristics  of  the  task  force  members  are  shown  in  Table  1. 

1995  Informal  Survey 

The  1995  Informal  Survey  was  distributed  by  the  Army  to  obtain  information  about 
AI/Robotics  jobs  that  would  be  helpful  in  creating  the  initial  lists  of  tasks  and  competencies.  The 
information  contained  answers  to  some  open-ended  questions  about  the  respondents’  positions, 
job  descriptions,  vita  curriculae,  and  descriptions  of  some  AI/Robotics  projects.  The  data  call 
results  also  identified  some  of  the  career  programs,  commands,  and  subcommands  that  had 
AI/Robotics  positions. 

From  this  information,  the  following  items  were  developed: 

•  a  preliminary  task  list 

•  a  preliminary  competency  list 

•  a  list  of  degree  programs  of  people  in  AI/Robotics 

•  a  list  of  institutions  that  offer  training  in  AI/Robotics 

•  demographic  information  about  the  respondents 

SME  Interviews 

All  four  task  force  members  were  interviewed  one-on-one  to  further  develop  the  task  and 
competency  lists.  In  each  interview,  the  SME  first  examined  the  current  lists  of  tasks  and 
competencies  and  suggested  additions,  deletions,  and  modifications.  The  SMEs  were  also  asked 
to  make  additions  to  the  lists  of  AI/Robotics  training  institutions  and  degree  programs  of  people 
in  AI/Robotics. 

Job  Analysis  'Workshops 

Project  staff  facilitated  two  one-day  job  analysis  workshops  which  used  a  group  consensus 
format.  The  participants  were  incumbents  in  AI/Robotics.  After  a  brief  orientation,  the  group 
made  additions,  deletions,  and  modifications  to  the  list  of  tasks.  Then  the  appropriateness  of  the 
task  categories  was  discussed.  The  group  also  made  additions,  deletions,  and  modifications  to  the 
list  of  competencies.  Then  the  appropriateness  of  the  competency  categories  was  discussed. 
Finally,  the  list  of  AI/Robotics  training  providers  was  edited  for  completeness  and  accuracy. 

The  first  workshop  was  held  January  25,  1996.  Five  civilians  (and  one  officer)  in  AI/Robotics 
attended.  The  second  workshop  was  held  February  21,  1996.  Eight  civilians  (and  one  officer)  in 
AI/Robotics  attended.  In  addition  to  the  activities  described  in  the  paragraph  above,  the 
participants  at  the  second  workshop  also  discussed  the  proposed  rating  scales  for  the  job  analysis 
survey.  As  a  result  of  this  discussion,  the  proposed  breadth  of  knowledge  required  and  depth  of 
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knowledge  required  scales  were  replaced  with  importance  and  level  of  knowledge  required  scales 
because  the  breadth  and  depth  scales  were  thought  to  be  confounded. 


Table  1.  Characteristics  of  the  Task  Force  Members 


SME  Characteristic 

Job  Title 

First  Task  Force  Member 

Operations  Research  Analyst 

Grade  Level 

GS-13 

Number  of  People  Supervised 

0 

Years  in  Current  Position 

11 

Years  of  Experience  in  AI 

7 

Current  Location 

U.S.  Army  Concepts  Analysis  Agency 

Gender 

Female 

Job  Title 

Second  Task  Force  Member 

Electronics  Engineer  /  Team  Leader 

Grade  Level 

GS-13 

Number  of  People  Supervised 

9 

Years  in  Current  Position 

1 

Years  of  Experience  in  AI 

9 

Current  Location 

Aberdeen  Test  Center,  Aberdeen  Proving  Ground 

Gender 

Male 

Job  Title 

Third  Task  Force  Member 

Operations  Research  Analyst 

Grade  Level 

GS-13 

Number  of  People  Supervised 

0 

Years  in  Current  Position 

8 

Years  of  Experience  in  AI 

6 

Current  Location 

U.S.  Army  Logistics  Integration  Agency,  New  Cumberland 

Gender 

Female 

Job  Title 

Fourth  Task  Force  Member 

Supervisory  Materials  Engineer 

Grade  Level 

GS-13 

Number  of  People  Supervised 

14 

Years  in  Current  Position 

5 

Years  of  Experience  in  AI 

3 

Current  Location 

U.S.  Army  Research  Lab,  Aberdeen  Proving  Grounds 

Gender 

Male 

1996  Data  Call 

The  1996  data  call  was  performed  to  obtain  an  exhaustive  list  of  Army  civilians  working  in 
AI/Robotics.  This  list  would  be  used  as  the  distribution  list  for  the  job  analysis  survey. 
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The  information  from  this  data  call  was  used  to  create  a  database  of  civilians  working  in 
AI/Robotics.  The  database  contained  the  following  information  about  each  respondent: 

•  Name 

•  Address 

•  Major  Command 

•  Major  Subordinate  Command 

•  Agency/Activity 

•  Mailing  Address 

•  Career  Program 

•  Grade 

•  Job  Series 

•  Percent  of  job  devoted  to  AI/Robotics  activities 
Job  Analysis  Questionnaire 

The  job  analysis  questionnaire  had  four  sections:  general  instructions,  questions  about  the 
respondent’s  position  and  training,  a  task  ratings  section,  and  a  competency  ratings  section.  In  the 
task  ratings  section,  respondents  rated  the  importance  of  37  tasks  and  both  the  importance  of  and 
level  of  knowledge  required  for  5 1  competencies. 

A  draft  of  the  job  analysis  survey  was  sent  to  sponsor  representatives  and  to  the  participants 
in  the  second  workshop  for  review.  A  few  minor  changes  were  made  based  on  the  reviewers’ 
suggestions. 

The  survey  materials  were  mailed  directly  to  each  person  on  the  list  (developed  from  Data 
Call  2)  of  Army  civilians  working  in  AI/Robotics.  The  materials  consisted  of  the  job  analysis 
questionnaire,  a  cover  letter,  and  a  stamped  return  envelope. 

Of  the  405  surveys  that  were  mailed,  240  were  returned.  This  represents  a  response  rate  of 
59%.  Only  171  respondents,  however,  performed  any  AI/Robotics  in  their  jobs.  Of  these,  121 
spent  more  than  5%  of  their  time  on  AI/Robotics  activities. 

Analysis  of  Survey  Results 

Based  on  the  responses  of  the  survey  respondents  who  spent  at  least  5%  of  their  time 
performing  AI/Robotics  tasks,  the  critical  AI/Robotics  tasks  and  the  important  competencies 
were  determined.  These  tasks  and  KSAs  were  reviewed  at  a  meeting  of  the  task  force.  The 
competencies  simulation  and  modeling  were  deleted  because  the  task  force  members  agreed  that, 
although  these  two  competencies  are  often  used  in  conjunction  with  AI/Robotics  competencies, 
they  themselves  were  not  AI/Robotics  competencies. 

Certification  Standards 

The  task  force  met  on  October  22,  1996,  to  finalize  the  list  of  competencies  and  begin  the 
development  of  the  certification  standards.  This  workshop  identified  the  courses  and  training  that 
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could  be  used  to  fulfill  the  requirements  for  each  competency.  One  of  the  task  force  members  had 
developed  a  list  of  many  of  the  available  courses  in  AI/Robotics.  The  workshop  participants 
determined  which  courses  could  be  used  to  fulfill  the  requirements  for  each  competency. 

The  task  force  and  three  other  AI/Robotics  incumbents  met  January  21-22,  1997.  The 
course  requirements  developed  at  the  previous  meeting  were  reviewed.  A  working  draft  of  the 
experience  standards  was  also  developed  at  this  meeting.  After  the  meeting,  this  draft  was  mailed 
to  the  workshop  participants  for  review. 

After  the  participants’  suggested  changes  were  incorporated  into  the  draft  of  the  experience 
standards,  definitions  for  each  of  the  competencies  were  developed.  All  the  draft  certification 
materials  were  then  sent  to  the  task  force  members  for  review.  HumRRO  then  modified  the 
materials  based  on  the  suggestions  by  task  force  members. 

The  task  force  met  a  final  time,  on  April  7,  1997,  to  review  and  finalize  the  certification, 
standards.  At  this  meeting,  one  of  the  core  competencies,  search  strategies,  was  moved  down  to 
the  level  of  a  supplemental  competency.  Minor  changes  were  made  to  other  portions  of  the 
certification  standards.  At  the  end  of  the  meeting,  all  the  task  force  members  concurred  with  the 
revised  certification  standards. 


Overview  of  the  Report 

The  remainder  of  this  report  discusses,  in  detail,  the  outcomes  at  each  step  of  the  project. 

Identification  of  Population  and  Positions.  The  results  of  two  data  calls  are  described  here.  The 
data  calls  obtained  information  about  the  civilians  working  in  AI/Robotics  in  the  Army. 

Frequency  analyses  for  several  position-related  variables,  such  as  job  series,  are  provided. 

Development  of  Lists  of  Tasks  and  Competencies.  This  section  describes,  in  detail,  how  the  lists 
of  tasks  and  competencies  for  the  job  analysis  survey  were  developed.  This  process  included  a 
data  call,  interviews  with  incumbents,  two  workshops  with  incumbents,  and  a  final  review  by  the 
task  force  members. 

Job  Analysis  Survey.  This  section  describes  both  the  survey  administration  process  and  the 
results  of  the  survey.  The  results  of  several  analyses  of  the  competency  and  task  ratings  are 
presented.  The  rationale  for  determining  which  competencies  are  important  and  which  tasks  are 
critical  is  also  explained. 

Construction  of  the  Course  Information  Base.  This  section  describes  the  development  of  the 
database  of  AI/Robotics  courses  that  can  be  used  to  meet  certification  requirements. 

Establishment  of  the  Task-Competency  Linkages.  This  section  describes  the  procedure  for  linking 
the  critical  tasks  with  the  important  competencies.  The  results  of  the  linking  survey  are 
presented. 
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Competency  Standards  Recommendations.  This  section  presents  the  recommended  certification 
standards  and  describes  the  final  steps  in  determining  these  standards.  The  report  concludes  with 
suggestions  for  implementing  the  certification  program. 

IDENTIFICATION  OF  POPULATION  AND  POSITIONS 

The  results  of  two  data  calls  are  described  here.  The  data  calls  obtained  information  about 
the  civilians  working  in  AI/Robotics  in  the  Army.  Frequency  analyses  for  several  position-related 
variables,  such  as  job  series,  are  provided. 

1995  Informal  Survey 

In  May,  1995,  the  project’s  sponsor  completed  an  informal  survey  of  civilians  working  in 
AI/Robotics.  A  questionnaire  was  distributed  to  a  small  sample  to  provide  initial  information 
about  Army  civilian  jobs  that  involved  AI/Robotics.  This  information  was  used  to  (a)  develop  an 
initial  list  of  tasks,  competencies,  training  sources,  and  formal  education  and  (b)  help  create  a 
distribution  list  for  a  comprehensive  data  call  which  would  attempt  to  identify  all  Army  civilians 
working  in  AI/Robotics. 

The  survey  was  sent  to  61  agency  points  of  contact  (POCs).  Individuals  returned  52 
completed  surveys;  41  sites  indicated  that  they  had  no  civilians  working  in  AI/Robotics.  The 
survey  results  show  that  (a)  most  of  the  respondents  were  in  Career  Program  16  (Non- 
Construction  Engineers  &  Scientists),  (b)  the  largest  percentage  of  respondents  (44%)  were  in 
pay  grade  GS-13,  and  (c)  for  half  of  the  respondents,  more  than  70%  of  their  duties  were  related 
to  AI/Robotics  (see  Tables  2-4). 


Table  2.  1995  Informal  Survey:  Percentage  of  Respondents  per  Career  Program 


Percentage  of 

Respondents  Career  Program 


81 

16 

Engineers  &  Scientists  (Non-Construction) 

9 

34 

Information  Mission  Area 

6 

18 

Engineers  &  Scientists  (Construction) 

2 

11 

Comptroller 

2 

14 

Contracting  &  Acquisition 

Note.  Percentages  are  based  on  a  total  of  50  responses. 
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Table  3.  1995  Informal  Survey:  Percentage  of  Respondents  per  Grade  Level 


Percentage  of 
Respondents 

Grade 

Level 

4 

GS-11 

28 

GS-12 

44 

GS-13 

20 

GS-14 

4 

GS-15 

Note.  Percentages  are  based  on  a  total  of  50  responses. 


Table  4.  1995  Informal  Survey:  Percentage  of  Duties  that  are  AI/Robotics-Related 


Percentage  of 
Respondents 

Percentage  of  Duties  that 
are  AI/Robitics-Related 

23 

100 

17 

80-99 

17 

60-79 

13 

40-59 

15 

20-39 

15 

1-19 

Note.  Percentages  are  based  on  a  total  of  47  responses. 


Many  of  the  respondents  provided  job  descriptions,  curriculae  vita,  information  about  some 
of  their  projects,  sources  of  training  and  education,  formal  degrees,  or  lists  of  AI/Robotics-related 
courses.  This  information  was  used  later  to  develop  preliminary  lists  of  tasks,  competencies,  and 
relevant  courses. 


1996  Data  Call 

In  December,  1995,  a  data  call  was  issued  as  part  of  the  present  project  to  help  develop  a 
comprehensive  list  of  civilians  in  AI/Robotics.  Initially,  sets  of  questionnaires  were  sent  to  the 
POCs  at  all  Major  Army  Commands.  In  addition,  POCs  were  identified  at  the  following 
subordinate  commands  within  the  HQDA  Major  Army  Command:  HQ  staff  and  principals. 
National  Guard  Bureau,  and  the  functional  chief  at  each  of  the  career  programs  identified  by  the 
1995  informal  survey. 
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When  the  initial  distribution  of  the  questionnaires  failed  to  produce  the  anticipated  volume 
of  responses,  alternative  modes  of  distribution  were  used.  About  two  weeks  past  the  deadline  for 
the  return  of  the  completed  surveys,  each  POC  was  phoned;  phone  calls  were  made  up  the  chain 
of  command  until  an  adequate  response  was  achieved.  In  some  cases,  the  final  POC  was  the 
commanding  general.  The  inadequate  initial  response  can  be  at  least  partially  attributed  to  a 
misunderstanding  by  some  POCS:  They  distributed  the  questionnaires  only  to  the  command’s 
primary  activity  rather  than  to  all  subordinate  activities  as  requested  by  the  cover  letter.  The 
difficulty  in  identifying  the  appropriate  POC  for  some  activities  also  hindered  the  efficient 
distribution  of  the  questionnaires.  The  data  call  was  completed  in  May,  1996. 

The  questionnaire  identified  405  civilians  who  perform  some  tasks  related  to  AI/Robotics. 
Seven  Career  Programs  and  24  Job  Series  were  represented.  For  reasons  discussed  in  the 
following  section,  the  results  from  the  data  call  are  probably  not  accurate.  There  are  probably  far 
fewer  than  405  civilians  performing  true  AI/Robotics  tasks,  and  the  headcounts  in  Figure  1  and 
Tables  5-8  are  probably  substantial  overestimates. 

About  one  quarter  of  the  respondents  (99  people) 
spent  at  least  half  of  their  time  doing  AI/Robotics  work; 
for  another  43%  (177  people),  AI/Robotics  was  less 
than  six  percent  of  their  job.  Figure  1  shows  a  detailed 
histogram.  The  shape  of  this  distribution  was  similar 
across  Job  Series,  Career  Programs,  and  Grade  Levels. 

The  one  exception  was  that  people  in  grade  levels  GS- 
14  and  GS-15  tended  to  spend  less  time  in  AI/Robotics 
activities  than  people  in  lower  grade  levels. 

Tables  5-8  and  Figure  1  show  respondent 
frequencies  by  Career  Program,  Job  Series,  and  Grade 
Level.  The  greatest  number  of  respondents  were  in 
Career  Program  16  -  Engineers  and  Scientists  (non¬ 
construction);  it  accounted  for  85%  of  the  respondents. 

The  most  common  Job  Series  was  855  -  Electronics 
Engineering.  Finally,  most  (61%)  of  the  respondents 
were  in  grades  GS-12  and  GS-13. 
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Figure  1.  1996  data  call:  Histogram  of 
percentage  of  work  related  to  AI/Robotics. 
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The  information  from  this  data  call  was  used  to  create  a  database  of  civilians  working  in 
AI/Robotics.  The  database  contained  the  following  information  about  each  respondent: 

•  Name 

•  Address 

•  Major  Command 

•  Major  Subordinate  Command 

•  Agency/Activity 

•  Mailing  Address 

•  Career  Program 

•  Grade 

•  Job  Series 

•  Percent  of  job  devoted  to  AI/Robotics  activities 


Table  5.  1996  Data  Call:  Frequencies  for  Career  Program 


Number  of 
Respondents 

Career  Program 

319 

16 

Engineers  &  Scientists  (Non-Construction) 

33 

34 

Information  Mission  Area 

8 

18 

Engineers  &  Scientists  (Construction) 

7 

11 

Comptroller 

5 

00 

none 

3 

13 

Supply  Management 

1 

24 

Transportation  Management 

1 

26 

Contracting  &  Acquisition 
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Table  6.  1996  Data  Call:  Frequencies 
for  Grade  Level 

Table  7.  1996  Data  Call:  Frequencies  for 
Major  Command 

Number  of 
Respondents 

Grade  Level 

Number  of 
Respondents 

Major  Command 

1 

GS-05 

339 

AMC 

1 

GS-07 

30 

HQDA 

4 

GS-09 

13 

TRADOC 

32 

GS-11 

9 

Corps  of  Engineers 

113 

GS-12 

4 

MEDCOM 

134 

GS-13 

8 

Others 

58  GS-14 

39  GS-15 


2  GM-13 

6  GM-14 

6  GM-15 

1  SES 

1  ST 


10 


Table  8.  1996  Data  Call:  Frequencies  for  Job  Series 


Number  of 
Respondents 

Job  Series 

80 

855 

Electronics  Engineering 

50 

1515 

Operations  Research 

40 

801 

General  Engineering 

37 

180 

Psychology 

34 

830 

Mechanical  Engineering 

32 

334 

Computer  Specialist 

27 

861 

Aerospace  Engineering 

21 

301 

Information  Systems  Manager 

14 

806 

Materials  Engineering 

14 

1550 

Computer  Science 

13 

1310 

Physics 

12 

1520 

Mathematics 

8 

850 

Electrical  Engineering 

7 

403 

Microbiology 

7 

896 

Industrial  Engineering 

3 

854 

Computer  Engineering 

2 

1529 

Mathematical  Statistics 

1 

856 

Electronics  Technician 

1 

858 

Biomedical  Engineering 

1 

893 

Chemical  Engineering 

1 

1320 

Metallurgy 

1 

1321 

Chemistry 

1 

2101 

Transportation  Specialist 

1 

2130 

Traffic  Management 

Job  Analysis  Survey 

A  job  analysis  survey  was  mailed  to  all  405  data  call  respondents.  (This  survey  will  be 
discussed  at  length  later.)  Although  all  of  the  data  call  respondents  indicated  that  their  jobs 
involved  at  least  some  activities  related  to  AI/Robotics,  29%  of  the  240  job  analysis  survey 
respondents  indicated  that  they  performed  no  work  in  AI/Robotics.  It  is  also  reasonable  to 
assume  that  many  of  those  who  failed  to  respond  to  the  job  analysis  survey  also  did  not  perform 
AI/Robotics  tasks. 
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There  are  at  least  three  possible  explanations  for  this  discrepancy.  First,  it  is  known  that 
several  data  call  questionnaires  were  filled  out  for  incumbents  by  other  people.  Some  of  these 
incumbents  may  not  have  done  any  work  in  AI/Robotics  (the  people  who  completed  their 
questionnaires  mistakenly  thought  that  they  did). 

Second,  it  is  possible  that  many  of  the  people  who  were  willing  to  answer  a  few  questions 
on  the  two-page  data  call  did  not  want  to  spend  the  time  to  complete  the  much  longer  job 
analysis  survey.  The  survey  was  16  pages  long  and  contained  152  questions.  Therefore,  some  of 
these  people  may  have  stated  that  they  did  no  work  in  AI/Robotics  to  avoid  answering  the  survey. 

Third,  it  seems  likely  that  some  people  may  have  discovered  that  they  did  not  perform  any 
of  the  tasks  listed  in  the  survey  and  concluded  that  they  do  not  work  in  AI/Robotics  after  all.  That 
is,  their  definition  of  AI/Robotics  was  narrower  for  the  job  analysis  survey  than  for  the  data  call. 
There  is  evidence  that  this  occurred.  First,  in  the  data  call,  AI/Robotics  activities  accounted  for 
less  than  three  percent  of  the  job  for  27%  of  the  respondents  and  less  than  six  percent  of  the  job 
for  43%  of  them.  Then,  in  some  of  the  returned  job  analysis  surveys,  the  respondent  had  rated 
only  the  first  few  tasks  (and  rated  each  of  them  as  not  being  performed)  and  had  also  filled  in  the 
circle  at  the  top  of  the  questionnaire  to  indicate  that  they  were  NOT  an  Army  civilian  working  in 
artificial  intelligence  or  robotics.  It  is  likely  that  these  people  (a)  started  filling  out  the  survey, 

(b)  realized  that  they  performed  none  of  the  AI/Robotics  tasks,  (c)  concluded  that  they  did  not  do 
any  AI/Robotics  activities  after  all,  and  (d)  filled  in  the  circle  at  the  top  of  the  questionnaire  to 
indicate  that  they  did  not  work  in  AI/Robotics. 

Thus,  although  the  data  call  identified  405  Army  civilians  performing  AI/Robotics  work, 
this  figure  appears  to  be  a  substantial  overestimate  of  the  actual  population.  The  job  analysis 
survey  identified  a  much  lower  figure  of  170  people  working  in  AI/Robotics,  and  even  this 
number  might  be  judged  to  be  an  overestimate  if  you  consider  that  it  includes  people  who  spend 
minimal  time  on  AI/Robotics  activities.  Analyses  for  the  job  analysis  survey  used  only  the  120 
(71%)  respondents  who  indicated  that  they  spend  5%  or  more  of  their  time  on  AI/Robotics  tasks. 
Thus,  the  population  actually  contains  a  minimum  of  120  positions  if  this  narrower  definition  is 
used  (i.e.,  spend  5%  or  more  of  the  time  on  AI/Robotics  tasks)  and  a  minimum  of  240  positions 
if  the  wider  definition  is  used  (i.e.,  spend  any  time  on  AI/Robotics  activities).  A  full  description 
of  the  sample  we  used  for  analysis  purposes  is  provided  later  in  this  report. 

DEVELOPMENT  OF  TASK  AND  COMPETENCY  LISTS 

The  lists  of  tasks  and  competencies  were  systematically  developed  via  the  following 
sequence: 

•  obtain  information  about  tasks  and  competencies  via  1995  Informal  Survey 

•  interview  incumbents  to  refine  lists  of  tasks  and  competencies 

•  conduct  workshops  to  refine  lists  of  tasks  and  competencies 
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Construction  of  Preliminary  Lists 


As  discussed  previously,  the  1995  informal  survey  of  civilians  working  in  AI/Robotics 
provided  information  about  the  jobs  of  52  people  working  in  AI/Robotics.  A  list  of  tasks  and 
competencies  was  developed  based  on  job  descriptions,  curricula  vitae,  and  descriptions  of 
projects  provided  by  the  survey  respondents. 

Individual  job  analysis  interviews  were  held  with  the  four  project  task  force  members.  All 
were  civilians  working  in  AI/Robotics  and  had  several  years  experience  in  AI/Robotics.  The 
characteristics  of  these  four  SMEs  were  shown  previously  in  Table  1.  The  first  interview  used  the 
lists  of  tasks  and  competencies  developed  from  the  1995  informal  survey  as  a  starting  point.  The 
lists  were  modified  during  each  interview.  These  updated  lists  were  then  used  to  begin  each 
subsequent  interview.  A  structured  interview  protocol  was  used  to  conduct  these  interviews. 

Each  interview  consisted  of  the  following  steps: 

•  Explain  the  format  and  purpose  of  the  interview. 

•  Explain  the  proper  format  of  task  and  competency  statements. 

•  Review  the  current  lists  of  tasks  and  competencies.  Add,  modify,  and  delete  task  and 
competency  statements. 

•  Ask  relevant  questions  from  the  list  of  job  analysis  interview  questions.  Use  the  answers 
to  modify  and  add  task  and  competency  statements. 

•  Review  the  lists  of  training  sources  and  relevant  educational  programs.  Add,  modify  and 
delete  items  in  these  lists. 

After  the  last  interview,  the  preliminary  lists  of  tasks  and  competencies  were  completed. 
These  lists  would  be  used  in  the  first  job  analysis  workshop. 

Job  Analysis  Workshops 

Two  job  analysis  workshops  were  conducted  to  develop  the  lists  of  tasks  and  competencies. 
The  two  workshops,  held  on  January  24-25  and  February  22-23, 1996,  each  lasted  V/2  days. 
There  were  five  SMEs  at  the  first  workshop  and  eight  SMEs  at  the  second.  In  addition,  an  officer 
from  the  Army  Artificial  Intelligence  Center  attended  both  workshops.  Each  task  force  member 
attended  one  workshop.  The  workshop  participants  were  chosen  by  the  sponsor  to  represent  a 
variety  of  AI/Robotics  positions.  The  characteristics  of  the  participants  are  shown  below  in  Table 
9.  There  were  1 1  white  males,  two  white  females,  and  one  African  American  male  at  the 
workshops. 

The  workshops  were  conducted  by  the  project  staff.  Each  workshop  began  with  an  overview 
of  the  entire  project  followed  by  a  synopsis  of  the  workshop.  Next,  the  SMEs  were  instructed  on 
how  to  write  task  and  competency  statements.  The  remainder  of  the  time  was  devoted  to 
developing  the  lists  of  tasks  and  competencies.  After  the  SMEs  read  the  task  list  to  themselves, 
they  gave  suggestions  for  deleting,  adding,  or  modifying  task  statements.  The  revised  list  was 
reviewed  and  further  refinements  were  made.  The  same  procedure  was  followed  for  the 
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competencies  as  well.  The  workshop  participants  also  reviewed  the  task  and  competency  rating 
scales  proposed  for  the  job  analysis  survey. 

The  lists  of  tasks  and  competencies  developed  via  the  interviews  were  used  as  the  starting 
point  in  the  first  workshop.  Similarly,  the  lists  revised  during  the  first  workshop  were  used  as  the 
starting  point  for  the  second  workshop.  Throughout  the  workshops,  managerial  tasks  that 
required  no  knowledge  of  AI/Robotics  were  excluded  from  consideration. 

JOB  ANALYSIS  SURVEY: 

DESCRIPTION  OF  THE  SAMPLE  AND  ADMINISTRATION  PROCEDURES 

After  the  second  job  analysis  workshop,  a  draft  of  the  job  analysis  questionnaire  was 
developed.  The  draft  contained  instructions,  biographical  questions,  and  the  lists  of  tasks  and 
competencies.  The  draft  was  sent  to  the  task  force  members  for  review.  Minor  changes  were 
made  based  on  the  reviewers’  comments. 

Job  Analysis  Questionnaire 

The  job  analysis  questionnaire  (see  Appendix  C)  had  four  sections:  general  instructions, 
questions  about  the  respondent’s  position  and  training,  a  task  ratings  section,  and  a  competency 
ratings  section.  In  the  task  ratings  section,  respondents  rated  the  importance  of  37  tasks  and  both 
the  importance  of  and  level  of  knowledge  required  for  5 1  competencies.  The  questionnaire 
contained  the  following  sections: 

•  General  Instructions 

•  Background  questions 

•  Task  rating  section.  Each  of  the  37  tasks  was  rated  on  how  much  it  was  a  part  of  the  job 
compared  with  the  other  AI/Robotics  tasks  performed  by  the  respondent. 

•  Knowledges  (i.e.,  competencies)  ratings  section.  Each  of  the  51  knowledges  was  rated  on 
two  things:  (a)  its  importance  relative  to  other  AI/Robotics  knowledges  and  (b)  the  level 
of  knowledge  required  for  successful  performance. 

•  Related  training  section.  Respondents  were  instructed  to  list  names  and  sources  of  the 
AI/Robotics  courses  they  had  taken. 

Administration  Procedures 

The  questionnaire  was  mailed  on  June  17,  1996,  directly  to  the  405  people  identified  by  the 
data  call  as  civilians  working  in  AI/Robotics  (and  for  whom  we  had  addresses:  five  of  the  405 
data  call  respondents  omitted  their  address).  Included  with  the  questionnaire  was  a  cover  letter 
from  the  Office  of  the  DISC4  signed  by  the  Director.  The  cover  letter  instructed  that  the 
questionnaire  be  mailed  directly  to  HumRRO  by  July  10,  1996  (i.e.,  about  four  weeks  later). 
Even  if  the  person  receiving  the  questionnaire  did  not  perform  any  AI/Robotics  activities,  the 
instructions  were  to  return  the  uncompleted  questionnaire  with  their  name  on  it.  Although  the 
respondents  put  their  names  on  the  questionnaires,  they  were  assured  that  their  results  would  be 
kept  confidential  and  not  used  to  evaluate  them  in  any  way.  One  week  before  the  deadline,  a 
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Table  9.  Characteristics  of  the  Job  Analysis  Workshop  Participants 


Job  Series 

Grade 

Years  in 
Current 
Position 

Years 
Experience 
in  AI 

Location 

806  -  Materials  Engineering 

13 

5 

3 

AMC,  Army  Research  Lab, 
Aberdeen  Proving  Ground 

? 

? 

? 

? 

Security  Asst.  Command 

? 

? 

? 

? 

AMC,  Army  Research  Lab, 
Aberdeen  Proving  Ground 

1530  -  Statistician 

13 

2 

12 

AMC,  Army  Research  Lab, 
Aberdeen  Proving  Ground 

1520  -  Mathematics 

13 

2 

12 

AMC,  Army  Research  Lab, 
Aberdeen  Proving  Ground 

830-  Mechanical  Engineering 

12 

4 

5 

MTMC,  Transportation 
Engineering  Agency 

? 

? 

? 

? 

Army  Research  Lab, 

Adelphi 

830-  Mechanical  Engineering 

11 

5 

USAFISA,  Ft.  Belvoir 

334  -  Computer  Specialist 

11 

5 

8 

IS  Command,  Army  Re-use 
Center 

806  -  Materials  Engineering 

14 

4 

3 

IS  Command,  Army  Re-use 
Center 

801  -  General  Engineering 

14 

? 

? 

AMC,  TECOM,  Aberdeen 
Proving  Ground 

855  -  Electronics  Engineering 

13 

1 

9 

AMC,  Aberdeen  Test 

Center,  Aberdeen  Proving 
Ground 

1515  -  Operations  Research 

13 

11 

7 

HQDA,  Chief  of  Staff, 

Army  Concepts  Analysis 
Agency 

15 


reminder  postcard  was  mailed  to  those  who  had  not  yet  returned  the  questionnaire.  The  postcard 
instructed  that  the  questionnaire  be  returned  as  soon  as  possible,  even  if  that  meant  going  past  the 
deadline. 


Description  of  the  Sample 

Of  the  405  surveys  that  were  mailed,  240  were  returned.  This  represents  a  respectable 
response  rate  of  59%.  Table  10  shows  the  details  concerning  the  survey  response  rate.  These 
statistics  are  complicated  by  the  large  number  of  respondents  who  stated  that  they  did  no  work  in 
AI/Robotics.  This  issue  was  discussed  in  detail  in  the  Identification  of  Population  and  Positions 
section  above. 


Table  10.  Job  Analyses  Survey  Response  Rates 


Number 

Percent  of 

Number  Mailed 

Mailed 

405 

100 

Returned  questionnaire  on  time 

240 

59 

Returned  questionnaire,  AI/Robotics 
work  is  at  least  some  part  of  job 

171 

42 

Returned  questionnaire,  AI/Robotics 
work  is  >  5%  of  job 

121 

30 

Returned  questionnaire  too  late  to  be 
included 

4 

1.0 

No  longer  at  mailing  address 

2 

0.5 
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Of  the  171  survey  respondents  working  in 
AI/Robotics,  only  121  spent  more  than  5  percent  of  their 
time  on  AI/Robotics  tasks  (see  Figure  2).  It  might  be 
argued  that  people  who  spend  only  a  small  proportion  of  -g 
their  time  performing  AI/Robotics  tasks  should  not  be  ■§ 
included  in  the  analyses  of  the  survey  results.  It  may  have  & 
been  difficult  for  these  people  to  make  accurate  ratings  £ 
on  such  a  small  part  of  their  jobs.  In  addition,  these  ° 

people  may  not  be  candidates  for  certification.  Therefore,  f 
analyses  were  performed  to  determine  whether  the  z 

ratings  differed  substantially  depending  on  the  amount  of 
time  spent  in  AI/Robotics  activities. 


For  these  analyses,  the  sample  was  split  into  12  9  19  29  39  49  59  69  79  89  95 

groups  based  on  the  amount  of  time  devoted  to  Percentage  of  Work  Related  to  AI/Robotics 

AI/Robotics.  The  frequency  distributions  of  the  12 

groups  were  compared  on  the  following  variables:  (a)  Figure  2.  Job  analysis  survey:  Histogram  of 
average  knowledge  importance  and  (b)  average  task  percentage  of  Al/Robotics-related  work. 

importance.  Tasks  not  performed  and  knowledges  not  required  were  assigned  a  score  of  zero. 
The  analyses  showed  that  the  mean  for  the  group  which  spent  less  than  five  percent  of  their  time 
in  AI/Robotics  activities  had  a  mean  much  lower  than  the  means  for  the  other  groups  (see  Table 
11). 
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Table  11.  Mean  Task  and  Knowledge  Ratings  by  Percentage  of  Time  Spent  in  AI/Robotics 
Activities 


Percentage  of  Time 
Spent  in  AI/Robotics 
Activities 

Number  of 
People  In 
Group 

Mean 

Task 

Importance 

Mean 

Knowledge 

Importance 

<5% 

50 

0.58 

0.85 

6-9 

12 

0.91 

1.46 

10-19 

18 

1.24 

1.60 

20-29 

18 

1.42 

1.43 

30-39 

12 

1.20 

1.24 

40-49 

5 

1.28 

1.54 

50-59 

7 

1.82 

1.70 

60-69 

10 

1.91 

1.92 

70-79 

13 

1.81 

1.81 

80-89 

9 

2.10 

1.99 

90-95 

9 

2.23 

1.76 

>95 

8 

2.01 

2.12 

Therefore,  it  was  decided  to  exclude  this  group  from  subsequent  analyses  of  the  knowledge 
and  task  ratings.  That  is,  the  analyses  would  include  only  the  121  people  who  spent  at  least  five 
percent  of  their  time  in  AI/Robotics  activities.  Frequency  ratings  are  reported  both  including  (N  = 
171)  and  excluding  (TV  =  121)  people  who  spent  less  than  five  percent  in  AI/Robotics  activities. 

The  biographical  variables  are  shown  in  Tables  12-21.  In  most  of  the  tables,  the  rows  are  in 
descending  rank  order  according  to  the  value  for  the  sample  of  people  that  spent  more  than  five 
percent  of  their  job  performing  AI/Robotics  tasks.  The  average  survey  respondent: 

•  works  in  the  Army  Research  Lab  in  AMC. 

•  is  in  the  Electronics  Engineer  job  series. 

•  is  in  the  Engineer  and  Scientist  (Non-Construction)  career  program. 

•  is  a  GS-1 1  or  GS-12. 

•  gets  most  of  his  or  her  training  from  reading,  on  the  job  training,  and  conferences. 

•  has  a  Master’s  degree. 

•  has  been  in  his  or  her  current  position  for  about  10  years. 

•  has  been  using  AI/Robotics  in  his  or  her  current  position  for  about  five  years. 

•  does  not  supervise  anyone. 
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Table  12.  Job  Analysis  Survey:  Frequencies  for  Career  Program 


Number  of  Respondents 

All 

AI/Robotics 

Respondents 

>  5%  of  Job 

Career  Program 

137 

95 

16 

Engineers  &  Scientists  (Non-Construction) 

11 

9 

34 

Information  Mission  Area 

3 

3 

13 

Supply  Management 

2 

2 

11 

Comptroller 

2 

2 

18 

Engineers  &  Scientists  (Construction) 

1 

1 

24 

Transportation  Management 

1 

1 

26 

Contracting  &  Acquisition 

9 

7 

Other 

5 

1 

Missing 

Table  13.  Job  Analysis  Survey:  Table  14.  Job  Analysis  Survey:  Frequencies  for 

Frequencies  for  Grade  Level _  Major  Command _ 


Number  of  Number  of 

Respondents _  Respondents 


AI/Robotics 

Grade 

AI/Robotics 

All 

>5%  of  Job 

Level 

All 

>5%  of  Job 

Major  Command 

2 

2 

GS-07 

123 

87 

AMC 

17 

13 

GS-09 

28 

20 

HQDA  Total 

53 

43 

GS-11 

6 

4 

ODCSLOG 

58 

40 

GS-12 

5 

3 

OPTEC 

14 

9 

GS-13 

3 

3 

CAA 

11 

5 

GS-14 

1 

1 

OCSPER 

5 

4 

GS-15 

1 

1 

ODISC4 

9 

6 

Other 

4 

1 

GM-13 

3 

2 

Missing  _ 

5 

2 

GM-14 

8 

6 

TRADOC 

2 

2 

GM-15 

4 

4 

Corps  of  Engineers 

4 

2 

Other 

4 

2 

Missing 
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Table  15.  Job  Analysis  Survey:  Frequencies  for  Job  Series 


Number  of  Respondents 

All  AI/Robotics 

Respondents  >  5%  of  Job  Job  Series 


26 

18 

855 

Electronics  Engineering 

22 

15 

1515 

Operations  Research 

16 

12 

334 

Computer  Specialist 

16 

11 

830 

Mechanical  Engineering 

14 

9 

180 

Psychology 

13 

8 

801 

General  Engineering 

10 

6 

861 

Aerospace  Engineering 

6 

6 

403 

Microbiology 

7 

6 

1550 

Computer  Science 

5 

5 

850 

Electrical  Engineering 

9 

5 

1310 

Physics 

6 

4 

301 

Information  Systems  Manager 

3 

2 

806 

Materials  Engineering 

3 

1 

896 

Industrial  Engineering 

1 

1 

2101 

Transportation  Specialist 

1 

1 

858 

Biomedical  Engineering 

2 

1 

1529 

Mathematical  Statistics 

11 

10 

2130 

Other 
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Table  16.  Job  Analysis  Survey:  Frequencies  for  Major  Subordinate  Command 


Number  of  Respondents 

All 

Al/Robotics 

Respondents 

>  5%  of  Job 

Major  Subordinate  Command 

46 

29 

Army  Research  Lab  (ARL) 

10 

10 

Army  Test  and  Evaluation  Command  (TECOM) 

12 

7 

Industrial  Engineering  Activity  (IEA) 

7 

4 

Industrial  Operations  Command  (IOC) 

6 

3 

Logistics  Support  Activity  (LOGSA) 

2 

2 

Missile  Command  (MICOM) 

3 

2 

Communications-Electronics  Command  (CECOM) 

44 

28 

Other 

41 

31 

Missing 

Table  17.  Job  Analysis  Survey:  Frequencies  for  Source  of  Training 


Number  of  Respondents 


All 

Respondents 

AI/Robotics 
>  5%  of  Job 

Source  of  Training/Education 

123 

93 

Reading  AI/Robotics  literature 

128 

98 

On-the-job  training 

86 

69 

Attending  conferences 

58 

49 

College  courses 

37 

34 

Private  providers  of  training  (e.g.,  NeuralWare) 

35 

33 

Government  providers  of  training:  Military  (e.g.,  Army 
Logistics  Management  College) 

5 

5 

Government  providers  of  training:  Non-military  (e.g.,  FBI) 

Notes.  N  =  171  for  All  Respondents  and  N  =  121  for  AI/Robotics  >  5%  of  Job.  Columns  do  not 
sum  to  these  totals  because  respondents  picked  all  training  sources  they  used. 
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Table  18.  Job  Analysis  Survey:  Frequencies  for  Highest  Level  of  Education  Achieved 


Number  of  Respondents 

All 

Respondents 

AI/Robotics 
>5%  of  Job 

Highest  Level  of  Education  Achieved 

7 

7 

Some  college 

4 

2 

Associate’s  degree  (2-year) 

20 

12 

Bachelor’s  degree  (4-year) 

28 

21 

Some  graduate  school 

66 

44 

Master’s  degree 

33 

24 

Doctoral  degree 

13 

11 

Missing 

Table  19.  Job  Analysis  Survey:  Frequencies  for  Time  in  Current  Position 


Number  of  Respondents 


All 

Respondents 

AI/Robotics 
>5%  of  Job 

Time  in  Current  Position 

4 

2 

Less  than  6  months 

5 

4 

6  months  to  less  than  one  year 

24 

17 

1  year  to  less  than  3  years 

21 

15 

3  years  to  less  than  5  years 

56 

35 

5  years  to  less  than  10  years 

49 

38 

10  or  more  years 

12 

10 

Missing 
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Table  20.  Job  Analysis  Survey:  Frequencies  for  Years  that  AI/Robotics  has  been  used  in  Current 
Position 


Number  of  Respondents 


All 

Respondents 

AI/Robotics 
>5%  of  Job 

Length  of  Time  that  AI/Robotics  has  been  used  in  Current 
Position 

15 

4 

Less  than  6  months 

8 

4 

6  months  to  less  than  one  year 

37 

25 

1  year  to  less  than  3  years 

32 

28 

3  years  to  less  than  5  years 

43 

32 

5  years  to  less  than  10  years 

18 

16 

10  or  more  years 

18 

12 

Missing 

Table  21.  Job  Analysis  Survey:  Frequencies  for  Supervisory  Level 


Number  of  Respondents 


All 

Respondents 

AI/Robotics 
>5%  of  Job 

Supervisory  Level  of  the  Respondent 

93 

73 

Not  a  supervisor 

45 

28 

Team  leader  (not  in  an  officially  designated  supervisory 
position) 

16 

7 

First  level  supervisor 

4 

3 

Second  level  supervisor 

1 

0 

Above  second  level  supervisor 

12 

10 

Missing 

ANALYSIS  OF  THE  JOB  ANALYSIS  SURVEY  RESULTS 

The  analysis  of  the  responses  to  the  job  analysis  survey  focused  on  the  importance  ratings 
for  the  37  tasks  and  the  51  knowledges.  As  stated  above,  these  analyses  included  only  the  121 
respondents  who  spent  more  than  five  percent  of  their  time  performing  AI/Robotics  tasks.  The 
purposes  of  the  analyses  were  to  determine  which  tasks  and  knowledges  were  important.  The 
rating  scales  for  the  tasks  and  knowledges  are  shown  below. 
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Task  Importance  Scale 
0  -  Not  performed 

1  -  Much  less  a  part  of  the  job  than  most  other  AI/Robotics  tasks 

2  -  Somewhat  less  a  part  of  the  job  than  most  other  AI/Robotics  tasks 

3  -  A  part  of  the  job  about  the  same  amount  as  most  other  AI/Robotics  tasks 

4  -  Somewhat  more  a  part  of  the  job  than  most  other  AI/Robotics  tasks 

5  -  Much  more  a  part  of  the  job  than  most  other  AI/Robotics  tasks 


Knowledge  Importance  Scale 

0  -  Not  required  at  all  to  perform  the  job. 

1  -  Much  less  important  than  most  other  knowledges 

2  -  Somewhat  less  important  than  most  other  knowledges 

3  -  About  the  same  importance  as  most  other  knowledges 

4  -  Somewhat  more  important  than  most  other  knowledges 

5  -  Much  more  important  than  most  other  knowledges 


Level  of  Knowledge  Required 

1  -  General  understanding 

2  -  Intermediate  level  of  knowledge 

3  -  Advanced  level  of  knowledge 


The  identification  of  the  important  tasks  and  knowledges  was  complicated  by  the  diverse 
nature  of  the  field  of  AI/Robotics.  Only  a  few  of  the  AI/Robotics  tasks  and  knowledges  listed  in 
the  survey  were  expected  to  be  relevant  for  any  one  person.  Thus  few  tasks  or  knowledges  were 
expected  to  receive  high  ratings. 

As  shown  below,  the  survey  responses  found  both  these  expectations  to  be  true.  About  half 
of  the  tasks  and  half  of  the  knowledges  on  the  survey  were  relevant  to  the  average  person’s  job. 
As  a  consequence,  every  single  task  had  a  mean  rating  across  the  respondents  as  being  a  part  of 
the  job  less  than  most  other  AI/Robotics  tasks;  similarly,  every  single  knowledge  had  a  mean 
rating  across  the  respondents  as  being  less  important  than  most  other  AI/Robotics  knowledges. 
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•  The  mean  number  of  tasks,  out  of  the  37  in  the  survey,  not  performed  =18. 

•  The  mean  number  of  knowledges,  out  of  the  5 1  in  the  survey,  not  required  =  25. 

•  The  mean  task  importance  rating  (see  the  rating  scale  above)  =  2.8. 

•  The  mean  knowledge  importance  rating  (see  the  rating  scale  above)  =  2.5. 

To  gain  other  perspectives  on  the  ratings,  four  variations  of  these  ratings  were  computed  for 
each  task  and  knowledge.  These  four  variations  are  described  below  for  the  knowledge 
importance  ratings.  Similar  variations  were  computed  for  the  level  of  knowledge  required  and 
task  importance.  When  computing  the  mean  rating  for  a  knowledge  across  the  respondents,  a 
respondent’s  score  was  designated  as  missing  if  he/she  did  not  rate  the  knowledge.  That  is,  these 
respondents  were  excluded  when  computing  the  mean  rating  for  that  knowledge. 

A  rating  of  zero  on  a  task  meant  that  the  respondent  did  not  perform  that  task;  a  rating  of 
zero  on  a  knowledge  meant  that  the  knowledge  was  not  required  at  all  to  perform  the  job.  These 
zero  ratings  were  dealt  with  in  two  different  ways  when  the  mean  ratings  were  computed  across 
the  respondents:  they  were  either  ignored  or  included.  When  zero  ratings  were  ignored,  a  rating 
of  zero  was  considered  to  be  missing:  respondents  who  gave  a  zero  rating  to  a  knowledge  were 
excluded  when  the  mean  was  computed  for  that  knowledge.  A  mean  knowledge  importance 
which  is  computed  this  way  answers  the  question:  “Among  the  respondents  who  use  this 
knowledge,  how  important  is  it?”  When  zero  ratings  were  included,  a  rating  of  zero  was  included 
in  the  computation  of  the  mean  for  that  knowledge.  A  mean  knowledge  importance  which 
is  computed  this  way  answers  the  question:  “What  is  the  average  importance  of  this  knowledge 
across  all  the  respondents?”  If  a  knowledge  is  required  by  only  a  few  people,  but  rated  as  being 
very  important  by  every  person  who  uses  it,  then  that  knowledge  would  have  a  high  mean 
importance  ignoring  zero  but  a  low  mean  importance  including  zero.  Means  were  computed  this 
way  for  both  tasks  and  knowledges. 

When  computing  the  means,  the  ratings  were  either  weighted  or  unweighted.  When 
computing  a  weighted  mean,  a  knowledge  was  assigned  a  score  equal  to  the  product  of  the  rating 
and  the  proportion  of  the  respondent’s  time  spent  in  AI/Robotics  activities.  This  weighting 
scheme  assumes  that  more  attention  should  be  paid  to  respondents  who  spend  a  lot  of  their  time 
in  AI/Robotics  activities. 

•  ignore  zero,  unweighted:  A  knowledge  Not  Required  was  given  a  score  of  missing.  A 
required  knowledge  was  assigned  a  score  equal  to  the  rating  assigned.  Thus  the  range  of 
possible  values  =  missing,  1-5. 

•  include  zero,  unweighted:  A  knowledge  Not  Required  was  given  a  score  of  zero.  A 
required  knowledge  was  assigned  a  score  equal  to  the  rating  assigned.  Thus  the  range  of 
possible  values  =  missing,  0-5. 

•  ignore  zero,  weighted:  A  knowledge  Not  Required  was  given  a  score  of  missing.  A 
required  knowledge  was  assigned  a  score  equal  to  the  product  of  the  rating  and  the 
proportion  of  the  respondent’s  time  spent  in  AI/Robotics  activities.  Thus  the  range  of 
possible  values  =  missing,  .075—4.875. 

•  include  zero,  weighted:  A  knowledge  Not  Required  was  given  a  score  of  zero.  A 
required  knowledge  was  assigned  a  score  equal  to  the  product  of  the  rating  and  the 
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proportion  of  the  respondent’s  time  spent  in  AI/Robotics  activities.  Thus  the  range  of 
possible  values  =  missing,  .075-4.875. 

One  more  perspective  was  gained  by  doing  separate  mean  importance  ratings  for  (a) 
respondents  who  spent  less  than  half  of  their  time  doing  AI/Robotics  and  (b)  respondents  who 
spent  more  than  half  of  their  time  doing  AI/Robotics.  All  perspectives  were  considered  when 
determining  the  important  tasks  and  the  critical  knowledges. 

In  general,  the  most  consideration  should  probably  be  given  to  ratings  that  are  unweighted, 
include  zero,  and  involve  people  who  spent  at  least  50%  of  their  time  in  AI/Robotics  activities. 

Descriptive  Statistics 


Task  Importance 

The  purpose  of  the  task  ratings  was  to  determine  the  critical  tasks  (these  are  listed  in  Table 
22).  These  critical  tasks  would  then  be  used  in  a  questionnaire  that  linked  the  competencies  to  the 
critical  tasks.  The  mean  task  importance  ratings  are  shown  in  Table  A-l,  and  their  ranks  are  in 
Table  A-2.  In  the  tables,  the  tasks  are  listed  in  order  of  their  mean  importance  ratings. 

Specifically  they  are  listed  in  order  of  the  mean  of  the  ranks  on  the  first  four  variables  listed  in 
the  table.  Seventeen  of  these  tasks  were  identified  as  being  critical. 

In  determining  which  tasks  were  critical,  the  following  factors  were  considered:  (a)  large 
jumps  in  the  ratings  between  two  tasks,  and  (b)  the  consistency  of  the  rankings  of  the  tasks  across 
all  the  variations  of  mean  importance.  In  addition,  more  attention  was  paid  to  the  ratings  from  the 
people  who  spent  at  least  50%  of  their  time  in  AI/Robotics,  the  unweighted  means,  and  the 
means  that  included  items  with  ratings  of  zero. 

Competencies 

Each  competency  was  rated  on  two  scales:  importance  and  level  of  knowledge  required.  The 
main  purpose  of  the  importance  ratings  was  to  help  identify  which  competencies  should  be 
included  in  the  certification  standards.  The  main  purpose  of  the  level  required  ratings  would  be  to 
help  the  certification  panel  determine  the  level  of  knowledge  that  a  certificant  should  have  in  a 
specific  competency.  It  would  also  be  useful  in  determining  which  competencies  should  be 
included  in  the  certification  standards. 

The  mean  knowledge  importance  ratings  are  shown  in  Table  A-3,  and  the  ranks  of  the 
means  are  in  Table  A-4.  In  the  tables,  the  competencies  are  listed  in  order  of  their  mean 
importance  ratings.  Specifically  they  are  listed  in  order  of  the  mean  of  the  ranks  on  the  first  four 
variables  listed  in  the  table. 

At  this  stage  in  the  project,  no  competencies  were  eliminated.  These  analyses  would  guide 
the  task  force’s  decision  regarding  which  competencies  to  include  in  the  certification  standards, 
additional  information  could  affect  the  decision. 
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Table  22.  List  of  Critical  Tasks 


1 .  Identify  and  assess  opportunities  to  apply  Al/robotics  to  improve  processes  and  systems. 

2.  Perform  overall  project  management  (e.g.,  monitor  budgets  and  timelines)  of  funded 
AI/Robotics  efforts. 

3.  Manage  daily  functions  of  funded  Al/robotics  efforts. 

4.  Read  research  articles,  reports,  and  others  sources  (e.g.,  internet  messages)  to  keep  abreast  of 
current  and  emerging  AI/Robotics  practices  and  technologies. 

5.  Oversee  contractors  performing  Al/robotics  work. 

6.  Seek  and  maintain  liaisons  with  academia/industry/govemment  to  keep  abreast  of  current 
and  emerging  AI/Robotics  practices  and  technologies. 

7.  Build  systems  (e.g.,  computer  programs)  using  Al/robotics  principles  and  techniques. 

8.  Select  available  knowledge  sources  (e.g.,  subject  matter  experts,  data). 

9.  Assist  users  in  refining  Al/robotics  system  requirements. 

10.  Evaluate  proposals  and  plans  for  Al/robotics  projects  and  programs. 

11.  Develop  Al/robotics  system  specifications  (e.g.,  for  environment,  software,  hardware,  and 
interface). 

12.  Identify,  evaluate,  and  recommend  procurement  of  specific  hardware  and  software  tools. 

13.  Evaluate  Al/robotics  software,  hardware,  and  tools. 

14.  Integrate  Al/robotics  components  into  systems. 

15.  Perform  knowledge  engineering  (e.g.,  interviewing  subject  matter  experts,  evaluating  data). 

16.  Attend  courses,  symposia,  seminars,  and  conferences  to  keep  abreast  of  current  and 
emerging  AI/Robotics  practices  and  technologies. 

17.  Test  and  evaluate  Al/robotics  components  and/or  systems. 
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Factor  Analysis 


In  order  to  gain  insight  into  the  dimensions  underlying  the  competencies,  a  factor  analysis 
was  performed.  The  analysis  used  a  principal  axis  extraction  and  oblique  Harris-Kaiser  rotation 
(allowing  the  factors  to  be  maximally  oblique).  It  was  decided  to  extract  four  factors  because  of  a 
discontinuity  between  the  fourth  and  fifth  eigenvalues. 

The  four  factors  were  Expert  Systems  &  Knowledge  Bases,  Modeling  and  Simulation, 
Computer  Science,  and  Robotics.  Table  23  shows  the  factor  pattern  matrix.  Tables  24-27  show 
each  competency  listed  within  the  factor  on  which  it  loaded  highest.  Tables  24—27  also  show  the 
mean  importance  ratings  and  ranking  for  each  competency. 

Within  each  factor,  the  competencies  were  put  into  two  or  three  groups.  In  determining 
which  competencies  to  place  in  each  group,  the  distributions  of  the  means  were  examined  for 
relatively  large  jumps  between  the  means  of  consecutively  ranked  knowledges.  Correlation 
analyses  showed  that  the  knowledges  tended  to  have  similar  ranks  on  all  of  the  importance 
measures. 

The  task  force  discussed  the  competencies  at  two  meetings  to  decide  which  competencies 
would  be  used  in  the  certification  process.  The  task  force  concurred  with  the  dimensions 
determined  by  the  factor  analysis.  They  considered  the  Systems  &  Knowledge  Bases  dimension 
to  represent  the  core  competencies  of  AI/Robotics  whereas  the  other  three  dimensions  contained 
supplemental  competencies.  That  is,  the  core  competencies  should  be  common  to  all  people 
working  in  AI/Robotics  whereas  supplemental  competencies  were  probably  relevant  only  to 
certain  types  of  positions. 


28 


Table  23.  Job  Analysis  Survey:  Factor  Loadings  for  Unweighted  Competency  Importance 
Ratings 


Overall 

Item  Rank 

Item 

KSA  Description 

Simulation  & 
Modeling 

Factor  Loadings 

Expert  Computer 

Systems  /  programming, 

Knowledge  software,  & 
Bases  hardware 

Robotics 

24 

48 

research  design  methods 

80 

-4 

-13 

7 

13 

50 

applied  mathematics  (e.g.,  calculus, 
algebra)  and  statistics 

78 

-6 

-1 

-4 

1 

47 

modeling 

77 

13 

-8 

1 

29 

49 

measurement 

76 

-18 

-10 

14 

4 

46 

simulation 

73 

5 

-1 

9 

21 

51 

optimization  techniques 

65 

9 

19 

-13 

5 

44 

conceptual  modeling  of  problems 
and  solutions 

64 

18 

18 

-27 

3 

9 

knowledge  bases 

-1 

86 

7 

-18 

11 

13 

knowledge  engineering 

1 

85 

2 

-27 

15 

25 

knowledge  representation 
techniques  and  principles 

0 

81 

17 

-28 

2 

1 

expert  systems 

5 

80 

-4 

-18 

47 

37 

case-based  reasoning  software 

-9 

76 

0 

4 

48 

15 

frame-based  systems 

-19 

76 

-1 

13 

35 

11 

Al  shells 

5 

75 

-4 

-17 

25 

7 

heuristic  algorithms 

19 

74 

-27 

4 

42 

5 

agent  architectures 

-17 

73 

-9 

19 

34 

38 

the  application  of  commercial  Al 
tools  (e.g.,  preprocessing  tools,  Al 
shells) 

8 

73 

-5 

-11 

7 

12 

methodologies  for  integrating  Al 
with  modeling,  optimization, 
mathematical,  statistical,  or  other 
computer  programs 

26 

69 

-2 

-14 

6 

10 

the  basic  principles  of  Al 

-2 

69 

2 

11 

45 

6 

genetic  algorithms 

5 

66 

-26 

17 

49 

36 

expert  system  blackboard  software 

-11 

66 

2 

4 

18 

20 

search  strategies 

16 

65 

11 

-4 

46 

14 

first-order  logic 

-4 

64 

0 

21 

37 

3 

path  planning 

6 

63 

-21 

29 

20 

24 

distributed  databases 

-17 

61 

40 

-14 

41 

4 

blackboard  theory 

-11 

59 

3 

24 

32 

2 

neural  networks 

20 

57 

-27 

28 

31 

8 

fuzzy  logic 

17 

56 

-19 

26 

14 

33 

principles  of  object-oriented 
programming 

-5 

51 

51 

-23 

9 

35 

knowledge  representation  tools  and 
arthitectures  (e.g.,  relational 
databases) 

9 

51 

48 

-43 

43 

28 

natural  language 

-19 

47 

10 

26 

51 

39 

machine  translation  of  languages 

0 

28 

16 

20 

table  continues  . . . 
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Overall 

Item  Rank 

Competency 

Simulation  & 
Modeling 

Factor  Loadings 

Expert  Computer 

Systems  /  programming, 

Knowledge  software,  & 
Bases  hardware 

Robotics 

10 

32 

computer  system  software 
component  interfaces 

-1 

-12 

96 

0 

19 

30 

computer  operating  systems 

-10 

-5 

83 

-6 

23 

31 

computer  system  hardware 
component  interfaces 

-6 

-23 

82 

29 

22 

29 

computer  system  architectures 

-16 

22 

71 

12 

16 

34 

high-level  computer  programming 
languages  (e.g.,  C,  LISP,  ADA) 

12 

5 

63 

-10 

8 

26 

software  engineering 

13 

21 

61 

-12 

17 

40 

real  time  programming 

14 

-5 

55 

20 

38 

45 

parallel  processing  techniques 

12 

13 

53 

13 

28 

43 

data  acquisition  hardware  and 
software 

40 

-30 

39 

26 

12 

41 

sensors 

12 

-29 

5 

79 

33 

18 

sensor  fusion 

5 

17 

-7 

76 

26 

21 

the  basic  principles  of  robotics 

3 

-18 

-2 

76 

44 

22 

robot  navigation 

-3 

-17 

18 

73 

39 

16 

control  theory 

-5 

21 

-5 

68 

40 

27 

computer  vision 

-6 

-11 

27 

68 

50 

19 

message  fusion 

-14 

48 

-10 

55 

27 

17 

embedded  systems 

-10 

38 

11 

53 

30 

42 

image  processing  software  and 
hardware 

26 

-10 

19 

53 

36 

23 

hybrid  systems 

-1 

41 

-15 

41 

Notes.  The  factor  loadings  indicate  the  extent  to  which  a  competency  is  related  to  each  of  the  four  factors. 

A  boldfaced  factor  loading  indicates  that  this  competency  can  be  considered  to  be  part  of  this  factor. 

Shading  is  added  to  enhance  readability;  it  does  not  signify  any  kind  of  grouping.  Overall  item  rank  is  based 
on  the  mean  importance  (unweighted,  include  zero)  for  AI  >  50%. 


The  task  force  made  the  following  decisions  concerning  the  list  of  competencies.  First,  the 
task  force  decided  to  eliminate  all  of  the  competencies  within  the  simulation  and  modeling 
dimension  except  for  conceptual  modeling  of  problems  and  solutions  (which  the  task  force 
concluded  is  important  in  AI/Robotics).  Although  simulation  and  modeling  are  often  used  in 
conjunction  with  AI/Robotics,  they  are  not  considered  to  be  part  of  AI/Robotics.  Both  of  these 
competencies  had  been  rated  highly.  Second,  the  task  force  concluded  that  some  of  the 
knowledges  related  to  computer  hardware  and  software  were  not  specifically  related  to 
AI/Robotics  and  thus  were  also  eliminated.  The  task  force  eliminated  mathematics,  statistics,  and 
measurement  for  two  reasons:  they  were  in  the  simulation  and  modeling  dimension  and  they 
were  not  specific  to  AI/Robotics. 
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Table  24.  Job  Analysis  Survey:  Mean  Competency  Importance  Ratings  for  Expert  Systems  & 
Knowledge  Bases 


Rank  Item 

Competency  Description 

AI  <  5% 

AI  5-50% 

AI  >  50% 

1 

1 

expert  systems 

1.20 

1.90 

2.72 

2 

9 

knowledge  bases 

1.02 

1.84 

2.67 

3 

33 

principles  of  object-oriented  programming 

1.00 

1.58 

2.55 

4 

25 

knowledge  representation  techniques  and 

principles 

1.02 

1.45 

2.53 

5 

10 

the  basic  principles  of  AI 

1.37 

1.92 

2.44 

6 

12 

methodologies  for  integrating  AI  with  modeling, 

optimization,  mathematical,  statistical,  or  other 

computer  programs 

1.08 

1.67 

2.44 

7 

35 

knowledge  representation  tools  and  architectures 

(e.g.,  relational  databases) 

1.10 

1.71 

2.40 

8 

13 

knowledge  engineering 

1.06 

1.68 

2.33 

9 

20 

search  strategies 

0.44 

1.39 

2.20 

10 

24 

distributed  databases 

0.98 

1.15 

2.15 

11 

7 

heuristic  algorithms 

0.76 

1.00 

1.79 

12 

8 

fuzzy  logic 

0.73 

1.30 

1.65 

13 

2 

neural  networks 

1.02 

1.44 

1.65 

14 

38 

the  application  of  commercial  AI  tools  (e.g., 

preprocessing  tools,  AI  shells) 

0.53 

1.13 

1.59 

15 

11 

AI  shells 

0.51 

1.36 

1.57 

16 

23 

hybrid  systems 

0.21 

1.10 

1.56 

17 

3 

path  planning 

0.39 

1.29 

1.56 

18 

4 

blackboard  theory 

0.27 

0.69 

1.37 

19 

5 

agent  architectures 

0.27 

0.62 

1.37 

20 

28 

natural  language 

0.41 

0.72 

1.29 

21 

6 

genetic  algorithms 

0.31 

0.94 

1.20 

22 

14 

first-order  logic 

0.41 

1.14 

1.13 

23 

37 

case-based  reasoning  software 

0.35 

0.79 

1.11 

24 

15 

frame-based  systems 

0.37 

0.73 

1.09 

25 

36 

expert  system  blackboard  software 

0.24 

0.65 

1.05 

26 

39 

machine  translation  of  languages 

0.39 

0.79 

0.75 

Notes .  Rank  is  based  on  the  AI  >  50%  mean.  Shading  is  added  to  enhance  readability;  it  does 
not  signify  any  kind  of  grouping.  Blank  rows  separate  sets  of  competencies  according  to  their 
importance. 
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Table  25.  Job  Analysis  Survey:  Mean  Competency  Importance  Ratings  for  Simulation  & 
Modeling 


Rank  Item 

Competency  Description 

AI  <  5% 

AI  5-50% 

AI  >  50% 

1 

47 

modeling 

3.26 

3.48 

3.76 

2 

46 

simulation 

3.04 

3.36 

3.62 

3 

44 

conceptual  modeling  of  problems  and  solutions 

2.43 

3.00 

3.18 

4 

50 

applied  mathematics  (e.g.,  calculus,  algebra)  and 
statistics 

2.83 

3.70 

2.76 

5 

51 

optimization  techniques 

2.09 

2.88 

2.65 

6 

48 

research  design  methods 

3.09 

2.88 

2.62 

7 

49 

measurement 

2.61 

2.69 

2.44 

Notes .  Rank  is  based  on  the  AI  >  50%  mean.  Blank  rows  separate  sets  of  competencies 
according  to  their  importance. 


Table  26.  Job  Analysis  Survey:  Mean  Competency  Importance  Ratings  for  Computer  Science 


Rank  Item 

Competency  Description 

AI  <  5% 

AI  5-50% 

AI  >  50% 

1 

32 

computer  system  software  component  interfaces 

3.09 

3.50 

3.46 

2 

26 

software  engineering 

2.64 

3.33 

3.39 

3 

34 

high-level  computer  programming  languages 
(e.g.,  C,  LISP,  ADA) 

2.64 

3.17 

3.36 

4 

31 

computer  system  hardware  component 
interfaces 

3.18 

3.39 

3.14 

5 

29 

computer  system  architectures 

2.91 

3.22 

3.07 

6 

40 

real  time  programming 

1.45 

2.50 

3.04 

7 

30 

computer  operating  systems 

3.27 

3.39 

3.00 

8 

45 

parallel  processing  techniques 

0.91 

2.61 

2.50 

9 

43 

data  acquisition  hardware  and  software 

2.45 

2.72 

2.25 

Notes .  Rank  is  based  on  the  AI  >  50%  mean.  Blank  rows  separate  sets  of  competencies 
according  to  their  importance. 
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Table  27.  Job  Analysis  Survey:  Mean  Competency  Importance  Ratings  for  Robotics 


Rank 

Item 

Competency  Description 

AI  <  5% 

AI  5-50% 

AI  >  50% 

1 

41 

sensors 

3.58 

3.06 

3.58 

2 

18 

sensor  fusion 

2.17 

2.29 

2.73 

3 

21 

the  basic  principles  of  robotics 

1.18 

2.45 

2.61 

4 

42 

image  processing  software  and  hardware 

1.73 

2.27 

2.58 

5 

17 

embedded  systems 

1.08 

2.29 

2.53 

6 

16 

control  theory 

2.08 

2.47 

2.40 

7 

27 

computer  vision 

0.83 

1.73 

2.32 

8 

22 

robot  navigation 

1.36 

1.91 

2.10 

9 

19 

message  fusion 

1.83 

1.61 

1.53 

Notes .  Rank  is  based  on  the  AI  >  50%  mean.  Shading  is  added  to  enhance  readability;  it  does 
not  signify  any  kind  of  grouping.  Blank  rows  separate  sets  of  competencies  according  to  their 
importance. 


Subgroup  Analyses 

Several  other  analyses  were  conducted  to  look  at  differences  between  subgroups.  Because  of 
the  small  sample,  few  meaningful  analyses  could  be  done.  The  subgroup  analyses  examined  the 
competency  rating  profiles.  They  used  the  unweighted  rating  measure  that  included  competencies 
with  zero  values.  The  profiles  are  shown  in  Figures  3-7. 

For  the  robotics  profile  chart,  people  who  did  little  work  in  this  area  were  excluded  (i.e., 
people  whose  sum  of  scores  on  the  robotics  competencies  was  9  or  less).  Similar  criteria  were 
used  to  exclude  people  from  the  profile  charts  for  computer  science  (i.e.,  people  whose  sum  of 
scores  on  the  computer  science  competencies  was  15  or  less),  and  simulation  and  modeling  (i.e., 
people  whose  sum  of  scores  on  the  simulation  and  modeling  competencies  was  1 1  or  less). 

The  groups  profiled  and  the  competencies  included  in  the  profiles  were: 

•  Competency  rating  by  the  amount  of  time  spent  in  AI/Robotics  activities  for  all 
competencies:  very  low  (<  5%)  vs.  low  (5-50%)  vs.  high  (>  50%) 

•  Competency  rating  by  the  amount  of  time  spent  in  AI/Robotics  activities  for  Expert 
Systems  /  Knowledge  Bases  competencies:  very  low  (<  5%)  vs.  low  (5-50%)  vs.  high  (> 
50%) 

•  Competency  rating  by  the  amount  of  time  spent  in  AI/Robotics  activities  for  Computer 
Science  competencies:  very  low  (<  5%)  vs.  low  (5-50%)  vs.  high  (>  50%) 

•  Competency  rating  by  the  amount  of  time  spent  in  AI/Robotics  activities  for  Robotics 
competencies:  very  low  (<  5%)  vs.  low  (5-50%)  vs.  high  (>  50%) 

•  Competency  rating  by  the  amount  of  time  spent  in  AI/Robotics  activities  for  Simulation 
and  Modeling  competencies:  very  low  (<  5%)  vs.  low  (5-50%)  vs.  high  (>  50%) 
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•  Competency  rating  by  the  importance  of  doing  basic  research  (task  #37):  none  (not 
performed)  vs.  minimal  (performed  much  less  than  other  AI/Robotics  tasks)  vs. 
significant  (performed  somewhat  less  to  much  more  than  other  AI/Robotics  tasks) 

The  profile  charts  show  that  the  patterns  of  the  competency  profiles  are  similar  among  the 
three  groups  that  were  formed  based  on  the  amount  of  time  spent  in  AI/Robotics  activities.  The 
elevations  of  the  profiles  did  differ,  however.  That  is,  people  who  spent  a  lot  of  time  in 
AI/Robotics  activities  tended  to  rate  the  knowledges  higher. 

The  competency  profile  chart  which  splits  the  sample  into  three  groups  based  on  the  amount 
of  basic  research  that  they  do  shows  that  the  profiles  look  very  similar,  overall,  for  the  more 
important  competencies.  For  the  less  important  competencies,  people  who  do  a  lot  of  basic 
research  considered  these  competencies  more  important  than  did  the  other  people. 


Core  Competency  Importance  by  Percent  of  Job  Devoted  to  Al 
4T 

! 


Note :  n  =  49  for  Very  Low,  n  =  65  for  Low,  and  n  =  56  for  High. 


Figure  3.  Profile  for  all  competencies  of  mean  importance  by  AI/Robotics  percentage. 
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Computer  Science  Competency  Importance 
by  Percent  of  Job  Devoted  to  Al 

High  At:  >50% 


Figure  4.  Profile  for  computer  science  of  mean  importance  by  AI/Robotics  percentage. 


Figure  5.  Profile  for  robotics  competencies  of  mean  importance  by  AI/Robotics 
percentage. 
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Modeling  and  Simulation  Competency  Importance 


Figure  6.  Profile  for  modeling  &  simulation  competencies  of  importance  by 
AI/Robotics  percentage. 


Competency  Importance  by  Amount  of  Basic  Research 
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Rank  of  Competency 


Note :  n  =  53  for  No  Basic  Research,  n  =  16  for  Minimal  Basic  Research  and  n  =  50  for  Significant  Basic  Research. 


Figure  7.  Profile  for  all  competencies  of  importance  by  basic  research  importance. 
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ESTABLISHMENT  OF  THE  TASK-COMPETENCY  LINKAGES 


A  competency-to-task  linkage  questionnaire  was  developed  to  ensure  a  close  linkage 
between  the  competencies  and  the  critical  tasks.  The  questionnaire  was  mailed  to  the  task  force 
members  with  written  instructions.  For  each  of  the  17  critical  tasks,  the  task  force  members  rated 
how  important  each  of  the  core  and  supplemental  competencies  were  in  performing  that  task  (the 
questionnaire  is  in  Appendix  D). 

The  raters  used  the  following  scale  to  rate  each  competency: 


0  not  needed  This  competency  is  not  needed  to  perform  the  task.  Having  this 
competency  would  make  no  difference  in  the  performance  of  the 
task. 


1  helpful 


This  competency  is  helpful  in  performing  the  task,  but  not 
essential.  The  task  could  be  performed  without  this  competency, 
but  the  task  would  be  more  difficult  or  time-consuming. 


2  essential 


This  competency  is  essential  to  the  performance  of  the  task. 
Without  this  competency,  you  would  not  be  able  to  perform  the 
task. 


?  don't  know 


If  you  are  unfamiliar  with  either  the  task  or  the  competency,  then 
write  a  question  mark  in  the  square. 


All  four  task  force  members  completed  the  questionnaire.  Six  of  the  core  competencies 
received  several  ratings  of  essential.  That  is,  each  of  these  six  competencies  were  considered 
essential  to  successfully  performing  some  critical  tasks.  This  was  true  for  every  rater.  Two  of  the 
raters  gave  the  core  competency,  search  strategies,  no  essential  ratings.  At  the  final  workshop, 
this  competency  was  changed  to  a  supplemental  competency  with  the  full  agreement  of  all  task 
force  members. 


All  of  the  supplemental  competencies  received  at  least  some  ratings  of  helpful.  Thus,  each 
of  the  supplemental  competencies  was  considered  helpful  for  performing  some  critical  tasks. 

CONSTRUCTION  OF  THE  COURSE  INFORMATION  BASE 


The  competency  standards  would  require  that  candidates  for  certification  provide  evidence 
that  they  had  acquired  a  competency.  Two  types  of  evidence  would  be  acceptable:  evidence  of 
training/education,  and  evidence  of  relevant  experience.  To  help  both  the  candidates  and  the 
certification  review  panel,  a  database  of  known  courses  in  AI/Robotics  was  developed. 

The  last  page  of  the  job  analysis  questionnaire  asked  respondents  to  list  the  names  and 
sources  of  all  their  AI/Robotics  courses.  One  of  the  task  force  members  collected  these 
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completed  pages  and  gathered  information  about  these  courses.  There  were  initially  over  250 
courses.  Courses  that  were  obviously  unrelated  to  the  competencies  were  eliminated,  as  were 
courses  for  which  information  could  not  be  gathered. 

The  task  force  met  on  October  22,  1996,  to  discuss  these  courses  and  link  them  to  the 
competencies.  The  meeting  began  with  a  list  of  100  courses.  Because  this  list  was  much  too  long 
to  link  to  the  32  competencies,  similar  courses  were  grouped  together.  One  course  in  each  group 
was  designated  as  the  prototype  course  for  that  group. 

Then  the  task  force  linked  each  of  the  20  prototype  courses,  via  consensus,  to  the 
competencies.  For  each  competency,  each  prototype  course  was  determined  to  provide  either  (a) 
no  credit,  (b)  partial  credit,  or  (c)  full  credit  towards  that  competency.  A  course  with  partial  credit 
could  be  combined  with  sufficient  experience  to  fulfill  the  requirements  for  a  competency. 
Similarly,  a  course  with  full  credit  would  completely  fulfill  the  requirements  for  that  competency. 

A  course  information  database  was  constructed  by  the  Army  Artificial  Intelligence  Center 
using  MS  Access.  For  each  course,  the  following  information  is  provided:  course  title,  list  of 
similar  courses,  the  prototype  course,  list  of  relevant  competencies  with  their  level  of  credit 
provided  by  the  course  (partial  or  full),  and  a  course  description.  This  course  information  is 
provided  in  the  certification  standards  (see  Appendix  E). 

COMPETENCY  STANDARD  RECOMMENDATIONS 

Process  for  Determining  Recommended  Standards 

After  the  job  analysis  results  were  reviewed  by  the  task  force,  the  certification  standards 
were  developed  via  four  meetings  and  a  mailout.  Parts  of  this  process  have  been  described  above. 
This  section  provides  an  integrated  description  of  the  process.  Parts  that  have  been  described  in 
detail  above  will  only  be  summarized  here. 

Selecting  the  Competencies  for  Certification 

The  results  of  the  job  analysis  survey  were  presented  to  the  task  force  September  25,  1996. 
On  September  30,  the  task  force  met  to  discuss  the  competencies.  The  competencies  within  each 
of  the  four  dimensions  obtained  via  the  factor  analysis  were  examined  separately:  Core, 
Simulation  and  Modeling,  Computer  Science,  and  Robotics. 

As  mentioned  previously,  all  of  the  competencies,  except  one  (conceptual  modeling  of 
problems  and  solutions)  within  the  simulation  and  modeling  dimension  were  discarded  because 
they  are  not  considered  part  of  AI/Robotics.  Some  of  the  computer  science  competencies  were 
dropped  for  the  same  reason. 

In  the  robotics  dimension,  all  of  the  competencies  were  kept  except  for  message  fusion  and 
embedded  systems.  Message  fusion  was  rated  much  lower  than  the  other  competencies,  and 
embedded  systems  was  determined  to  be  redundant.  Computer  vision  and  image  processing  were 
combined  to  form  a  single  competency. 
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In  the  computer  science  dimension,  all  the  competencies  were  determined  to  be  not 
specifically  related  to  AJ/Robotics  except  for  high-level  computer  programming  languages. 

Thus,  only  one  competency  from  this  dimension  was  kept. 

In  the  core  dimension,  the  five  lowest-rated  competencies  were  discarded.  In  addition,  agent 
architectures  was  deleted  because  it  was  redundant,  and  distributed  databases  was  deleted 
because  it  was  considered  not  to  be  an  area  of  AI/Robotics.  Speech  recognition  /  synthesis  and 
machine  learning  were  added  because  they  were  considered  important  by  the  task  force  but  were 
not  on  the  survey. 

The  task  force  rearranged  the  retained  competencies  into  two  categories:  core  competencies 
and  supplemental  competencies.  The  first  category  represented  competencies  that  would  be  used 
in  most  types  of  AI/Robotics  applications,  whereas  the  second  category  included  competencies 
that  would  likely  be  used  only  in  specialized  applications.  The  task  force  decided  that  certificants 
should  possess  all  of  the  core  competencies  and  at  least  three  of  the  supplemental  ones. 

Linking  Courses  to  Competencies 

At  the  next  workshop,  held  October  22,  1996,  the  task  force  reviewed  the  course 
information  and  linked  the  courses  to  the  competencies.  Twenty  prototype  courses  were  linked  to 
the  competencies  via  consensus.  Each  course  was  identified  as  giving  partial  or  full  credit  for  a 
competency.  The  task  force  also  developed  a  general  description  of  the  level  of  performance  and 
capabilities  in  AI/Robotics  that  the  certificants  should  be  demonstrated  to  be  certified. 

Defining  Standards  for  Experience 

The  task  force  and  three  additional  Army  civilians  working  in  AI/Robotics  developed  most 
of  the  standards  for  experience  at  a  two-day  workshop  held  November  20-21,  1996.  They 
developed  the  outline  for  general  standards  for  experience  and  combinations  of  experience  and 
course  work.  Examples  of  sufficient  and  insufficient  experience  were  also  developed  for  each  of 
the  core  competencies. 

After  the  meeting,  HumRRO  refined  the  standards  and  sent  a  mailout  (on  February  20, 

1997)  to  the  task  force  members  for  review.  The  task  force  members  reviewed  the  standards  and 
suggested  a  few  minor  changes.  As  part  of  the  mailout,  the  task  force  members  also  completed  a 
questionnaire  to  link  the  competencies  to  the  critical  tasks. 

Finalizing  the  Standards 

The  task  force  met  one  more  time,  on  April  7,  1997.  The  sponsor  had  written  definitions  for 
the  competencies.  Prior  to  the  meeting,  the  task  force  members  had  read  the  definitions  and 
offered  suggestions.  The  definitions  were  reviewed  at  the  meeting,  and  minor  modifications  were 
made. 
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The  changes  to  the  definitions  elicited  a  couple  of  changes  to  the  competencies.  First,  A1 
shells  was  discarded  because  it  was  considered  redundant.  Next,  the  two  competencies 
concerning  knowledge  representation  were  combined  into  one  competency. 

Search  strategies  was  changed  from  a  core  to  a  supplemental  competency.  After  observing 
that  this  competence  was  considered  helpful  but  not  essential  to  performing  critical  tasks,  the 
task  force  members  concurred  that  it  should  be  considered  supplemental.  Based  on  the 
definitions  of  the  competencies,  minor  changes  were  made  to  the  wording  in  some  of  the 
competency  titles. 

The  recommended  Standards  for  the  Certification  of  Civilians  in  the  U.S.  Army  in  Artificial 
Intelligence  are  in  Appendix  E. 

Recommendations  for  Implementing  the  Certification  Standards 

Issues  are  bound  to  surface  when  the  new  certification  program  is  instituted.  For  example, 
no  set  of  standards  can  be  completely  objective  and  cover  all  situations.  Therefore,  some 
subjective  decisions  will  have  to  be  made.  Some  guidelines  are  provided  below  that  should  help 
both  the  applicants  and  the  review  panel  in  the  certification  process.  These  guidelines  were 
reviewed  and  accepted  by  the  task  force  at  the  final  project  meeting. 

Guidelines  for  Certification  Candidates 

It  is  important  that  certification  candidates  be  given  sufficient  information  to  allow  them  to 
understand  the  certification  process,  the  competency  standards,  and  procedures  for  preparing 
their  application  packages.  To  do  this,  a  candidate  handbook  should  be  developed  that  includes 
the  following  items: 

•  Purpose  of  the  certification  process 

•  Significance  of  the  credential  (i.e.,  what  it  means  and  what  it  does  not  mean) 

•  Full  description  of  the  requirements  for  certification 

•  Detailed  instructions  for  preparing  submission  packages 

•  Timeline  for  submitting  and  evaluating  packages 

•  Description  of  the  process  that  will  be  used  to  evaluate  the  submission  packages 

•  Who  to  contact  with  questions 

Much  of  the  information  for  the  handbook  can  be  drawn  from  the  competency  standards.  It 
would  be  hard  to  imagine  providing  too  much  information  to  candidates,  and  too  little 
information  will  result  in  candidates  preparing  packages  that  have  too  much  or  too  little  detail  for 
the  reviewers. 

Guidelines  for  the  Review  Panel 

The  competency  standards  (see  Appendix  D)  discuss  how  the  submission  packages  for 
Al/robotics  specialty  certification  candidates  should  be  evaluated.  It  is  recommended  that  each 
submission  package  be  reviewed  by  three  persons.  Each  reviewer  should  evaluate  a  package 
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independently  and  then  the  three  pass/fail  decisions  collated.  If  all  three  reviewers  do  not  agree 
on  the  overall  pass/fail  decision,  the  reviewers  should  meet  to  review  the  package  as  a  group  and 
reach  a  consensus  decision. 

To  assure  the  fairness  and  integrity  of  the  certification  process,  it  is  critical  that  reviewers 
adhere  closely  to  the  evaluation  guidelines.  This  is  particularly  important  if  a  reviewer’s  internal 
standards  of  competence  tend  to  be  higher  or  lower  than  those  reflected  in  the  guidelines. 
Adherence  to  the  guidelines  will  also  help  to  assure  that  reviewers  focus  on  evidence  associated 
with  the  relevant  competencies  and  ignore  other  factors  (e.g.,  neatness)  when  evaluating  the 
submission  packages.  If  there  is  insufficient  information  on  which  to  evaluate  a  candidate  on  one 
or  more  competencies,  the  review  panel  must  request  additional  information  or  fail  the  candidate. 
Reviewers  should  not  infer  information  that  is  not  provided. 
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APPENDIX  A.  Tables  of  Ratings  Results  from  Job  Analsis  Survey 


A-  1 


Table  A-l.  Job  Analysis  Survey:  Mean  Task  Importance 
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Notes.  Lower  rank  values  reflect  more  important  tasks.  The  final  rank  is  the  mean  of  all  four  ranks  in  the  row.  In  the  ignore  zero  columns,  the  mean  score  was  set 
to  missing  for  a  task  if  the  respondent  rated  it  as  being  not  performed.  In  the  include  zero  columns,  the  mean  score  was  set  to  zero  for  a  task  if  the  respondent 
rated  it  as  being  not  performed.  For  the  weighted  columns,  the  ratings  were  multiplied  by  the  proportion  of  time  the  respondent  spent  in  AI/Robotics  activities. 
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Notes.  Lower  rank  values  reflect  more  important  knowledges.  The  final  rank  is  based  on  the  mean  of  the  four  ranks  for  all  respondents  (i.e.,  ranks  for  unweighted 
include  zero,  unweighted  ignore  zero,  weighted  include  zero,  weighted  ignore  zero).  In  the  ignore  zero  columns,  the  mean  score  was  set  to  missing  for  a 
knowledge  if  the  respondent  rated  it  as  being  not  required.  In  the  include  zero  columns,  the  mean  score  was  set  to  zero  for  a  knowledge  if  the  respondent  rated  it 
as  being  not  required.  For  the  weighted  columns,  the  ratings  were  multiplied  by  the  proportion  of  time  the  respondent  spent  in  AI/Robotics  activities. 
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APPENDIX  B.  Data  Call  Questionnaire 
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Survey  for 

Army  Civilians  Working  in 
Artificial  Intelligence  or  Robotics 

Please  complete  this  questionnaire  if  you  are  an  Army  civilian  who  develops  and/or  manages 
artificial  intelligence  or  robotics  projects. 

Supervisors:  Please  complete  a  survey  for  each  unencumbered  (vacant)  artificial  intelligence/ 
robotics  position  in  your  office.  Print  “unencumbered”  in  the  “Name:”  block  below  for  an  unencumbered 
position. 

PRIVACY  ACT  STATEMENT 

Authority:  5  USC  SECTION  301,  Executive  Order  9397,  AR  340-21,  and  AR  61 1-3. 

Principal  Purpose:  To  determine  which  positions  have  activities  related  to  the  development  of  artificial  intelligence  or 
robotics. 

Routine  Uses:  Names  or  other  individual  identifiers  of  persons  completing  this  survey  will  be  treated  confidentially.  Data  will 
be  used  to  identify  persons  working  in  artificial  intelligence  or  robotics. 

Disclosure:  Completion  of  this  survey  is  voluntary.  However,  the  quality  of  the  Army  civilian  Artificial  Intelligence  Specialty 
Program  is  dependent  upon  employee  cooperation  in  completing  this  questionnaire. 


Complete  the  following  information  about  yourself.  Please  print. 
Name: 

(print  “unencumbered”  if 
position  is  currently  vacant) 

Major  Army  Command: 

Major  Subordinate 
Command: 

Agency/Activity : 

Mailing  Address: 

/complete  address:  include 
office  symbol  and  zip  code) 


Career  Program: 

Grade: 
Job  Series: 


Survey  for  Army  Civilians  Working  in  Artificial  Intelligence  or  Robotics 
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For  the  purposes  of  this  survey,  the  definition  of  artificial  intelligence  is: 


AI  is  a  branch  of  science  dedicated  to  the  development  of  computational  programs  that  exhibit  intelligent 
behaviors  and  actions  such  as  problem  solving,  learning,  and  understanding  language.  AI  techniques  imitate 
human  thought,  actions,  or  decision  making  processes  or  produce  results  similar  to  these  processes. 

For  the  purposes  of  this  survey,  the  definition  of  robotics  is: 

Robotics  is  a  branch  of  science  dedicated  to  the  study,  development,  and  application  of 
mechanical/electrical  systems  that  perform  tasks  involving  sensing,  computation,  and  action.  These 
mechanical/electrical  systems  can  be  teleoperated  (human  interface  is  used  for  control),  autonomous 
(independent  of  real-time  human  control),  or  semi-autonomous.  Military  applications  of  Army  robots 
generally  fall  into  two  categories:  tactical  and  industrial. 


1.  Given  the  above  definitions,  what  percentage  of  the  time  on  your  job  involves  tasks  related  to 
artificial  intelligence  and/or  robotics?  Circle  the  appropriate  percentage  below. 

none  <3%  3-5%  6-25%  26-50%  51-74%  75-94%  95-97%  >97% 


2.  List  any  areas  of  artificial  intelligence  or  robotics  in  which  you  are  involved  that  are  not  covered  by 
the  definition  above. 


*  End  of  Questionnaire  - 
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APPENDIX  C.  Job  Analysis  Questionnaire 


C-  1 


DEPARTMENT  OF  THE  ARMY 
OFFICE  OF  THE  SECRETARY  OF  THE  ARMY 
107  ARMY  PENTAGON 
WASHINGTON  DC  20310-0107 


Office.  Director  of  Information 
Systems  for  Command.  Control. 
Communications.  &  Computers 


SAIS-EDD 


S:  10  July  1996 
17  June  1996 


MEMORANDUM  FOR  DEPARTMENT  OF  THE  ARMY  CIVILIAN  ARTIFICIAL 

INTELLIGENCE  PROFESSIONALS 


SUBJECT:  Army  Civilian  Artificial  Intelligence  (AI)  Job  Analysis  Survey 


1 .  Your  name  was  identified  through  an  Army-wide  population  survey  as  an  Army  civilian 
currently  performing  duties  in  AI  and/or  robotics.  You  are  requested  to  complete  and  return  the 
attached  “Job  Analysis  Inventory  for  Civilians  Working  in  Artificial  Intelligence  and  Robotics” 
(enclosure  1). 

2.  The  “Job  Analysis  Inventory  for  Civilians  Working  in  Artificial  Intelligence  and  Robotics”  is  a 
critical  part  of  an  Army-wide  effort  to  create  a  one-of-a-kind  civilian  specialty  program  for  Army 
civilians.  In  order  to  fully  exploit  the  potential  of  Al/robotics,  it  is  imperative  for  the  Army  to 
identify,  train,  and  manage  a  talent  pool  of  individuals  with  Al/robotics  skills.  The  Army  has  a 
program  to  do  this  with  its  military  personnel  -  the  AI  Civilian  Specialty  Program  will  allow  the 
Army  to  do  the  same  with  its  civilian  personnel. 

3.  AI  is  a  unique  set  of  skills  not  restricted  to  a  given  civilian  career  program  (CP)  or  occupational 
series.  As  such,  this  program  will  involve  participation  by  the  Army  CPs  and  use  their  CP  training 
and  referral  systems.  It  will  identify  and  provide  AI  professional  development  and  training 
opportunities,  establish  an  AI  Specialty  Advisory  Council,  and  provide  a  voluntary  certification 
program  for  Army  civilians  with  AI  skills. 

4.  Instructions  for  completing  the  survey  are  contained  within  the  survey.  If  you  have  any 
questions  about  completing  the  survey,  please  contact  Dr.  Gordon  Waugh  at  the  Human 
Resources  Research  Organization  (HumRRO).  (703)  706-5666.  Completed  surveys  must  be 
returned  in  their  sealed  envelopes  NLT  10  July  1996.  Dr.  Waugh’s  address  is: 

Dr.  Gordon  Waugh 

Human  Resources  Research  Organization 
66  Canal  Center  Plaza,  Suite  400 
Alexandria,  VA  22314 
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SAIS-IDD 

SUBJECT:  Anny  Civilian  Artificial  Intelligence  (AI)  Job  Analysis  Survey 


5.  The  0DISC4  points  of  contact  are  Mrs.  Mary  Campbell  at  (703)  695-9304/DSN  225-9304  or 
MAJ  Bill  Atkinson  at  DSN  224-6904/(703)  614-6904. 
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Job  Analysis  Inventory  for 
Army  Civilians  Working  in 
Artificial  Intelligence  or  Robotics 


PRIVACY  ACT  STATEMENT 

Authority:  5  USC  SECTION  301,  Executive  Order  9397,  AR  340*21.  and  AR  61 1-3. 

Principal  Purpose:  To  determine  the  tasks  performed  and  knowledge  required  by  Army  civilians  working  in  Artificial  Intelligence 
and  Robotics. 

Routine  Uses:  Names  or  other  individual  identifiers  of  persons  completing  this  survey  will  be  treated  confidentially.  Data  will  be 
used  to  determine  the  similnrities  «nd  differences  in  usks  performed  »nd  knowledge  required  *mon8^™y  d^UUns 

Artificial  Intelligence  and  Robotics.  This  information  will  aid  in  creaUng  a  voluntary  certification  program  for  Army  civilians 

working  in  Artificial  Intelligence  and  Robotics. 

Disclosure:  Completion  of  this  survey  is  voluntary.  However,  the  quality  of  the  Army  civilian  Artificial  Intelligence  and  Roboucs 
Specialty  Program  is  dependent  upon  employee  cooperation  in  completing  this  questionnaire. 


Organization  of  Inventory 


This  inventory  is  divided  into  three  sections: 


Section  I  -  Background  Information  requests  information  about  your  position  and  your  job 
related  background  which  will  help  to  ensure  that  a  representative  group  of  job 
incumbents  has  been  surveyed  and  allow  examination  of  differences  in  job 
requirements  across  various  types  of  AI/Robotics  positions. 

Section  II  -  Job  Tasks  contains  a  list  of  tasks  related  to  AI/Robotics.  You  will  be  asked  to  read 
each  task  statement  and  indicate  how  much  the  task  is  a  part  of  your  job. 


Section  III  •  Knowledges  contains  a  list  of  knowledge  areas  related  to  AI/Robotics.  You  will  be 
asked  to  read  each  knowledge  statement  and  indicate  how  important  it  is  in  your  job 
and  the  level  of  the  knowledge  required  to  perform  your  job. 

Section  IV  -  Related  Training  requests  that  you  list  the  courses  that  have  helped  you  prepare  for 
your  AI/Robotics  work. 


General  Instructions  •  Read  CarefullyI _ _ _ _ _ 

This  is  not  a  test  —  it  is  a  survey.  Your  responses  to  this  survey  will  be  used  to  (a)  provide 
information  about  the  tasks  performed  in  AI/Robotics  jobs  and  the  knowledge  required  to 
perform  these  tasks,  (b)  help  create  a  voluntary  certification  program  for  Army  civilians  working 
in  AI/Robotics.  and  (c)  help  determine  the  criteria  and  procedures  for  the  training  and 
recruitment  of  AI/Robotics  personnel. 

This  survey  is  confidential.  It  will  not  be  used  to  evaluate  you  in  any  way. 

Do  not  be  concerned  if  there  are  many  tasks  and/or  knowledges  that  are  irrelevant  to  your 
position.  This  inventory  was  designed  to  cover  all  positions  in  AI/Robotics:  thus,  it  is  unlikely  that 
every  task  or  knowledge  statement  will  apply  to  your  current  position. 
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General  Instructions  -  ( Continued [ 


•  Respond  only  in  terms  of  your  current  position,  not  your  previous  positions  or  those  of  your 
subordinates. 

•  Think  of  your  job  as  you  actually  perform  it,  not  as  it  is  described  in  your  position  description 
or  as  you  think  it  should  be  performed. 

•  If  you  have  any  questions,  please  contact  the  person  at  your  location  responsible  for  the 
administration  of  this  inventory  or  call  Dr.  Gordon  Waugh  (703)  706-5666  at  the  Human 
Resources  Research  Organization  (HumRRO)  in  Alexandria,  Virginia. 

•  When  you  have  completed  the  inventory,  place  it  in  the  stamped  pre-addressed 
envelope  provided,  seal  it,  and  mail  it.  The  envelope  provided  is  addressed  to: 

Gordon  Waugh 
HumRRO 

66  Canal  Center  Plaza,  Suite  400 
Alexandria,  VA  22314 


For  the  purposes  of  this  survey,  the  definition  of  artificial  intelligence  is: 

...a  branch  of  science  dedicated  to  the  development  of  computational  programs  that 
exhibit  intelligent  behaviors  and  actions  such  as  problem  solving,  learning,  and 
understanding  language.  AI  techniques  imitate  human  thought,  actions,  or  decision 
making  processes  or  produce  results  similar  to  these  processes. 

For  the  purposes  of  this  survey,  the  definition  of  robotics  is: 

...a  branch  of  science  dedicated  to  the  study,  development,  and  application  of 
mechanical/electrical  systems  that  perform  tasks  involving  sensing,  computation, 
and  action.  These  mechanical/electrical  systems  can  be  teleope rated  ( human 
interface  is  used  for  control),  autonomous  (independent  of  real-time  human 
control),  or  semi-autonomous.  Military  applications  of  Army  robots  generally  fall 
into  two  categories:  tactical  and  industrial. 
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Section  I. 
Background 


NOTE: 

If  you  are  NOT  an  Army  civilian  working  in 
artificial  intelligence  or  robotics  as  defined  below, 
mark  here 

and  return  you  survey  uncompleted. 


Name: 


Mariana  Instructions 

CORRECT  MARK 


'  INCORRECT  MARKS 

n/X  •  • 

•  Use  a  No.  2  pencil. 

•  Fill  in  the  circle  completely. 

•  Erase  cleanly  any  marks  you  wish  to  change. 

•  Do  not  make  any  stray  marks  on  this  form. 


1.  What  percentage  of  the  time  on  your  job  involves  tasks 
related  to  artificial  intelligence  and/or  robotics? 


4.  What  job  series  are  you  in? 


none 

<5% 

50-59 

6-9 

60-69 

10-19 

70-79 

20-29 

80-89 

30-39 

90-95 

40-49 

>95 

2.  What  is  your  present  pay  grade? 

GS-7  GM-13 

GS-9  GM-14 

GS-11  GM-15 

GS-12 

GS-13  SES 

GS-14  ST 

GS-15 

Other  (please  specify)  _ 


3.  What  is  your  Career  Program? 

11  -  Comptroller 

13  -  Supply  Management 

16  -  Engineers  &  Scientists  (non-constiuction) 

18  -  Engineers  &  Scientists  (construction) 

24  -  Transportation  Management 
26  -  Manpower  and  Force  Management 
34  -  IMA 

Other  (please  specify) : 


180  -  Psychology 

301  -  Information  Systems  Manager 
334  -  Computer  Specialist 
403  -  Microbiology 
801  -  General  Engineering 
806  -  Materials  Engineering 
830  -  Mechanical  Engineering 
850  -  Electrical  Engineering 

854  -  Computer  Engineering 

855  -  Electronics  Engineering 

856  -  Electronics  Technician 
858  -  Biomedical  Engineering 
861  -  Aerospace  Engineering 
893  •  Chemical  Engineering 
896  -  Industrial  Engineering 
1310  •  Physics 

1320  •  Metallurgy 

1321  -  Chemistry 

1515-  Operationsd  Research 
1529  -  Mathematics  and  Statistics 
1550  -  Computer  Science 
2101  -  Transportation  Specialist 
2130  -  Traffic  Management 
Other  (please  specify) 
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Section  i. 
Background 


5.  Is  HQDA  (Headquarters,  Department  of  the  Army) 
your  Army  major  command  (MACOM)? 

No  »>  SKIP  TO  QUESTION  7  - . 

Yes 

6.  What  is  your  HQDA  activity? 

Concepts  Analysis  Agency  (CAA) 

Deputy  Chief  of  Staff  for  Logistics 
(ODCSLOG) 

Deputy  Chief  of  Staff  for  Personnel 
(ODCSPER) 

Office  of  the  Director  of  Information 
Systems  for  C4  (ODISC4) 

Operational  Tert  and  Evaluation  Command 
(OPTEC) 

T otal  Army  Personnel  Command 
Other  (please  specify): 


SKIP  TO  QUESTION  9 


7.  What  is  your  present  Army  major  ^ _ I 

command  (MACOM)? 

Criminal  Investigation  Command  (CIDC) 

Military  District  of  Washington  (MDW) 

Military  Traffic  Management  Command  (MTMC) 
OCONUS:  Command  = 
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8.  What  is  you  present  major  siiwrdinate  command? 

US  Army  Aviation  and  Troop  Command  (ATCOM) 

US  Army  Chemical  and  Biological  Defence  Command  (CBDCOM) 
US  Army  Combined  Arms  Center  (CAC) 

US  Army  Combined  Arms  Support  Center  (CASCOM) 

US  Army  Communications-Electronics  Command  (CECOM) 
USAMC  Executive  Director  for  Conventional  Ammunition  (EDCA) 
USA  Industrial  Engineering  Activity  (IEA) 

US  Army  Industrial  Operations  Command  (IOC) 

US  Army  Information  Systems  Software  Center  (ISSC) 

US  Army  Information  System  s  Engineering  Command  (ISEC) 
USAMC  Logistics  Support  Activity  (LOGSA) 

US  Army  Missile  Command  (MICOM) 

US  Army  Research  Laboratory  (ARL) 

US  Army  Security  Assistance  Command  (USASAC) 

US  Aimy  Simulation,  Training,  and  Instrumentation  Command 
(STRICOM) 

US  Army  Soldier  Systems  Command  (SSCOM) 

US  Army  Test  and  Evaluation  Command  (TECOM) 

I  Corps 
III  Corps 
V  Coips 

XVIII  Airborne  Corps 
Other  (please  specify): 


9.  Which  of  the  following  sources  of  training'education  have  you  used  to  team 
about  Al/Robotics?  (Marie  all  that  apply) 


US  Army  Corps  of  Engineers  (USACE) 

US  Army  Forces  Command  (FORSCOM) 

US  Army  Information  Systems  Command  (ISC) 
US  Army  Intelligence  and  Security  Command 
(INSCOM) 

US  Army  Materiel  Command  (AMC) 

US  Army  Medical  Command  (MEDCOM) 

US  Army  Training  and  Doctrine  Command 
(TRADOC) 

Other  (please  specify): 


Reading  Al/Robotics  literature 
On-the-job  training 
Attending  conferences 
College  courses 

Private  providers  of  training  (e.g.,  NeuralWare) 
Government  providers  of  training:  Military  (e.g.,  Aimy 
Logistics  Management  College) 

Government  providers  of  training:  Non-military  (e.g.,  FBI) 


C  -  7 


Section  I. 
Background 


10.  What  is  your  highest  level  of  education? 

High  school  diploma  or  equivalent 
Some  college 

Associate’s  degree  (2-year), 
specify  area  of  study  - 


Bachelor’s  degree  (4-year), 
specify  area  of  study  _ 


Some  graduate  school, 
specify  area  of  study 


13.  What  level  supervisor  are  you? 

Not  a  supervisor 

Team  leader  (not  in  an  officially  designated  supervisory 
position) 

First  level  supervisor 
Second  level  supervisor 
Above  second  level  supervisor 


Master’s  degree, 
specify  area  of  study 


Doctoral  degree, 
specify  area  of  study 


11.  How  long  have  you  been  in  your  present  position? 

Less  than  6  months 
6  months  to  less  than  1  year 
1  year  to  less  than  3  years 
3  years  to  less  than  5  years 

5  years  to  less  than  1 0  years 
10  or  more  years 

12.  How  long  have  you  used  Al/Robotics  in  your  present  position? 

Less  than  6  months 

6  months  to  less  than  1  year 
1  year  to  less  than  3  years 
3  years  to  less  than  5  years 
5  years  to  less  than  10  years 
10  or  more  years 
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Section  II. 
Job  Tasks 


This  section  contains  a  list  of  job  tasks  related  to  AI/Robotics  that  was  developed  through  extensive 
consultation  with  personnel  in  AI/Robotics.  You  will  note  that  the  range  of  tasks  is  very  broad  and 
there  may  be  many  tasks  that  are  not  relevant  to  your  current  position. 

To  familiarize  yourself  with  the  list,  you  should  first  read  all  of  the  task  statements  once  through. 
Next,  read  each  task  statement  in  turn,  and  rate  how  much  the  task  is  a  part  of  the  AI/Robotics  portion 
of  your  job  using  the  following  scale: 


A  Part  of  the  Job 


0  -  Not  performed 

1  -  Much  less  a  part  of  the  job  than  most  other  AI/Robotics  tasks 

2  -  Somewhat  less  a  part  of  the  job  than  most  other  AI/Robotics  tasks 

3  *  A  part  of  the  job  about  the  same  amount  as  most  other  AI/Robotics  tasks 

4  •  Somewhat  more  a  part  of  the  job  than  than  most  other  AI/Robotics  tasks 

5  •  Much  more  a  part  of  the  job  than  than  most  other  AI/Robotics  tasks 


Note  that  you  are  rating  each  task  relative  to  other  AI/Robotics  tasks  that  you  perform;  thus  rating  a 
task  “1”  does  not  mean  that  it  is  not  a  part  of  the  job,  just  that  it  is  not  as  much  a  part  of  the  job  as  other 
AI/Robotics  tasks  for  which  you  are  responsible. 

In  your  ratings  of  the  extent  to  which  each  task  is  a  part  of  the  job,  consider  how  often  you  perform  the 
task,  the  amount  of  time  you  spend  performing  the  task,  and  any  other  factors  you  think  are  related  to 
how  much  a  task  is  a  part  of  the  job.  Obviously,  the  tasks  you  do  from  day  to  day  vary  with  the 
projects  on  which  you  work.  Therefore,  think  about  your  job  over  time  when  making  these  ratings. 
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Section  II.  I 

Job  Tasks  I 

A  Part  of  the  Job: 

O-Not  performed. 

1  •  Much  less  a  part  erf  the  Job  than  most  rther  AVRobotlcs  tasks 

2  •  Somewhat  less  a  part  of  the  job  than  most  other  Al/Robcdcs  tasks 

3  *  A  part  of  the  job  about  the  same  amount  as  most  other  Al/Robctocs  tasks 

4  *  Somewhat  more  a  part  of  the  job  than  most  other  AVRobotlcs  tasks 

5  -  Much  more  a  part  of  the  job  than  most  other  AVRobotlcs  tasks 

A  PART  OF 
THE  JOB 


r*L 


% 

v* 


X%. 


i  i  i  i  i  i 


1.  Identify  training  needs  and  sources  for  Al/robotics  personnel. 

0  1  2  3  4  5 

2.  Develop/update  Al/robotics  workshops  and  seminars. 

3.  Develop/update  formal  Al/robotics  training  courses  (e.g.,  classroom  training). 

4.  Teach  Al/robotics  courses. 

5.  Perform  on-the-job  training  for  co-workers  in  the  use  of  Al/robotics  tools  and  techniques. 

6.  Advise  academia,  government,  and  industry  on  Al/robotics  topics. 

7.  Seek  and  maintain  liaisons  with  academia/industry/govemment  to  keep  abreast  of  current  and  emerging 
Al/robotics  practices  and  technologies. 

8.  Give  oral  presentations  on  Al/robotics  at  symposia,  conferences,  and  workshops. 

9.  Attend  courses,  symposia,  seminars,  and  conferences  to  keep  abreast  of  current  and  emerging  Al/robotics 
practices  and  technologies. 

10.  Read  research  articles,  reports,  and  other  sources  (e.g.,  internet  messages)  to  keep  abreast  of  current  and 
emerging  Al/robotics  practices  and  technologies. 

11.  Identify  and  assess  opportunities  to  apply  Al/robotics  to  improve  processes  and  systems. 

12.  Develop  written  or  oral  proposals  to  obtain  funds  for  Al/robotics  projects  and  programs. 

13.  Evaluate  proposals  and  plans  for  Al/robotics  projects  and  programs. 

14.  Identify,  evaluate,  and  recommend  outside  providers  for  Al/robotics-related  services. 

15.  Identify,  evaluate,  and  recommend  procurement  of  specific  hardware  and  software  tools. 

16.  Perform  overall  project  management  (e.g.,  monitor  budgets  and  timelines)  of  funded  Al/robotics  efforts. 

17.  Manage  daily  functions  of  funded  Al/robotics  efforts. 

18.  Write  technical  papers,  reports,  and  journal  articles  on  Al/robotics  efforts. 
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Section  II. 
Job  Tasks 


O-Not  performed. 

1  •  Much  less  a  part  of  the  Job  than  most  other  Al/Robcfk*  tasks 

2  -  Somewhat  less  a  part  of  the  job  than  most  ether  AVRobctics  tasks 

3  -  A  part  of  the  job  about  the  same  amount  as  most  other  Al/Robotocs  tasks 
4 -Somewhat  more  a  part  of  the  job  than  most  other  AYRobctlcs  tasks 

5  -  Much  more  a  part  of  the  job  than  most  other  AJ/Robotlcs  tasks 


A  PART  OF 
THE  JOB 


l  l  l  l  l  l 


19.  Give  briefings  on  Al/robotics-related  efforts  (e.g.,  to  higher  command  levels). 

0  1  2  3  4  5 

20.  Develop  and  revise  statements  of  work  related  to  Al/robotics. 

21.  Oversee  contractors  performing  Al/robotics  work. 

22.  Oversee  staff  performing  Al/robotics  work. 

23.  Conduct  basic  research  in  Al/robotics. 

24 .  Select  available  knowledge  sources  (e.g.,  subject  matter  experts,  data). 

25.  Develop  Al/robotics  system  specifications  (e.g.,  for  environment,  software,  hardware,  and  interface). 

26.  Perform  knowledge  engineering  (e.g.,  interviewing  subject  matter  experts,  evaluating  data). 

27.  Assist  users  in  refining  Al/robotics  system  requirements. 

28.  Build  systems  (e.g.,  computer  programs)  using  Al/robotics  principles  and  techniques. 

29.  Develop  tools  and  methodologies  to  assist  in  testing  Al/robotics  systems. 

30.  Evaluate  Al/robotics  software,  hardware,  and  tools. 

31.  Integrate  Al/robotics  components  into  systems. 

32.  Develop  knowledge  representations. 

33.  Identify  Al/robotics  re-use  opportunities. 

34 .  Test  and  evaluate  Al/robotics  components  and/or  systems. 

35.  Transition  Al/robotics  system  operation  knowledge  from  design  team  to  end  user. 

36.  Maintain  Al/robotics  systems  (e.g.,  update  database,  fix  hardware  and  software). 

37.  Upgrade  Al/robotics  systems  (e.g.,  add  new  capabilities). 
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Section  ill. 
Knowledges 


This  section  contains  a  list  of  knowledges  related  to  AI/Robotics  that  was  developed  through  extensive 
consultation  with  personnel  in  AI/Robotics.  You  will  note  that  the  range  of  knowledges  is  very  broad 
and  there  may  be  many  knowledges  that  are  not  relevant  to  your  current  position. 

To  familiarize  yourself  with  the  list,  you  should  first  read  all  of  the  knowledges  once  through.  Next, 
read  each  knowledge  in  turn,  and  rate  the  relevance  of  the  knowledge  to  performing  the  AI/Robotics 
tasks  in  your  job  using  the  following  scales: 

Importance 


0  •  Not  required  at  all  to  perform  the  AI/Robotics  tasks  on  this  job. 

1  -  Much  less  important  than  most  other  AI/Robotics-related  knowledges. 

2  •  Somewhat  less  important  than  most  other  AI/Robotics-related  knowledges. 

3  -  About  the  same  importance  as  most  other  AI/Robotics-related  knowledges. 

4  -  Somewhat  more  important  than  most  other  AI/Robotics-related  knowledges. 

5  -  Much  more  important  than  most  other  AI/Robotics-related  knowledges. 


©  If  you  rate  the  importance  of  a  knowledge  as  0-  Not  Required  et  sll  to  perform  the  job, 
then  do  not  rate  the  knowledge  on  the  Level  of  Knowledge  Required  scale. 


Level  of  Knowledge  Required 


1  -  General  understanding 

2  -  Intermediate  level  of  knowledge 

3  -  Advanced  level  of  knowledge 


When  making  your  Importance  ratings,  it  will  help  to  think  about  whether  or  not  you  could  accomplish 
important  AI/Robotics  job  tasks  without  a  particular  knowledge.  Many  knowledges  might  be  “nice  to 
know”  but  are  not  really  necessary  for  successful  job  performance.  Similarly,  when  making  your 
ratings  of  Level  of  Knowledge  Required,  consider  the  minimum  level  of  knowledge  required  for 
successful  performance  of  AI/Robotics  tasks,  not  the  level  that  would  be  “nice  to  have.” 
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Section  III. 
Knowledges 


Important; 

LeveL 

0  *  Not  required  at  all  to  perform  the  job. 

1  -  Much  lest  Important  than  most  other  knowledges 

2  -  Somewhat  less  Important  than  most  other  knowledges 

3  -  About  the  same  Importance  as  most  other  knowledges 

4  -  Somewhat  more  Important  than  most  other  knowledges 

5  •  Much  more  Important  than  most  other  knowledges 

Reauired: 

1  •  General  understanding 

2  •  Intermediate  level  of  knowledge 

3  -  Advanced  level  of  knowledge 

Knowledge  of... 

1.  expert  systems. 


2.  neural  networks. 


3.  path  planning. 


4.  blackboard  theory. 


5.  agent  architectures. 


6.  genetic  algorithms. 


7.  heuristic  algorithms. 


8.  fuzzy  logic. 


9.  knowledge  bases. 


10.  the  basic  principles  of  Al  theory. 


11.  Al  shells. 


12.  methodologies  for  integrating  Al  with  modeling,  database,  optimization,  mathematical, 
statistical,  or  other  computer  programs. 


13.  knowledge  engineering. 


14.  first-order  logic. 


15.  frame-based  systems. 


16.  control  theory. 


17.  embedded  systems. 


18.  sensor  fusion. 


19.  message  fusion. 


IMPORTANCE 


LEVEL  OF 
KNOWLEDGE 
REQUIRED 


•f  % 

I  I  I  I  I 


I  1  I  I  I 


C-  14 


Section  III. 
Knowledges 


Importance: 

WBBSSSSKM 

0  •  Not  required  at  all  to  perform  the  job. 

1  -  Much  less  Important  than  most  other  knowledges 

2  -  Somewhat  less  Important  than  most  other  knowledges 

3  *  About  the  same  Importance  as  most  other  knowledges 

4  -  Somewhat  more  Important  than  most  other  knowledges 

5  *  Much  more  Important  than  most  other  knowledges 

Required;  1 

1  -  General  understanding 

2  •  Intermediate  level  of  knowledge 

3  *  Advanced  level  of  knowledge 

Knowledge  of... 

20.  search  strategies. 

21.  the  basic  principles  of  robotics. 


22.  robot  navigation. 


23.  hybrid  systems. 

24.  distributed  databases. 


25.  knowledge  representation  techniques  and  principles. 

26.  software  engineering. 

27.  computer  vision. 


28.  natural  language. 


29.  computer  system  architectures. 


30.  computer  operating  systems. 


31 .  computer  system  hardware  component  interfaces. 


32.  computer  system  software  component  interfaces. 


33.  principles  of  object-oriented  programming. 


34 .  high-level  computer  programming  languages  (e.g.,  C,  LISP,  ADA). 


35.  knowledge  representation  tools  and  architectures  (e.g.,  relational  databases). 


36.  expert  system  blackboard  software. 


37.  case-based  reasoning  software. 

38.  the  application  of  commercial  Al  tools  (e.g.,  preprocessing  tools,  Al  shells). 

39.  machine  translation  of  languages. 


LEVEL  OF  "1-2 
KNOWLEDGE 
IMPORTANCE  REQUIRED 

i  i  i  i  i  i  i  i  i  i 


C-  15 


Section  III. 
Knowledges 


Importance: 

Level  of  Knowl 

edae 

0  -  Not  required  at  all  to  perform  the  job. 

1  *  Much  less  Important  than  most  other  knowledges 

2  •  Somewhat  less  Important  than  most  other  knowledges 

3  *  About  the  same  Importance  as  most  other  knowledges 

4  -  Somewhat  more  Important  than  most  other  knowledges 

5  *  Much  more  Important  than  most  other  knowledges 

Required: 

1  -  General  uiderstandng 

2  ■  Intermediate  level  of  knowledge 

3  *  Advanced  level  of  knowledge 

Knowledge  of. . . 

40.  real  time  programming. 


LEVEL  OF  ii 
KNOWLEDGE 
IMPORTANCE  REQUIRED 

%  W#w  ^ 

I  I  I  I  I  I  1  I  I  I 


1  2  3  4  5  1  2  3 


42 .  image  processing  software  and  hardware. 


43.  data  acquisition  hardware  and  software. 

4  4 .  conceptual  modeling  of  problems  and  solutions. 


45 .  parallel  processing  techniques. 

46.  simulation. 


47.  modeling. 


4  8 .  research  design  methods. 


50.  applied  mathematics  (e.g.,  calculus,  algebra)  and  statistics. 

51.  optimization  techniques. 


Additional  Knowledges 

In  the  space  below,  please  list  any  important  At/Robotics  knowledges  you  need  to  perform  your  job  that  did  not 
appear  in  the  list. 


End  of  Survey  -  Thanks  for  your  time. 
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Section  IV. 
Related  Training 


Please  list  the  courses  you  have  taken,  as  part  of  your  training  or  education,  that  have  helped  you  prepare 
for  the  AI/Robotics  requirements  of  your  job.  Include  only  courses  directly  related  to  AI/Robotics.  For 
example,  you  should  exclude  courses  in  math  and  physics  unless  they  focus  on  AI  or  Robotics.  The  first 
line  below  is  an  example  of  how  you  should  list  each  course. 

Course  Title  Source  of  Course 


Introduction  to  AI 


MIT 
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Directions  for  Completing  the  Task-to-Competency  Rating  Form 

The  left  side  of  the  Task-to-Competency  Rating  Form  displays  the  competencies  that  will  be  used  for 
AI/Robotics  certification.  The  numbered  columns  to  the  right  of  the  competencies  represent  the  tasks 
that  are  most  important  for  people  doing  AI/Robotics  according  to  the  survey  results.  The  accompanying 
sheet  shows  the  description  of  each  task. 

Your  job,  for  each  task  (i.e.,  each  numbered  column),  is  to  decide  whether  each  competency  is  either  not 
needed,  helpful,  or  essential  to  performing  the  task  properly. 

Please  follow  these  steps: 

1 .  Print  your  name  next  to  the  word  Name:  ~>  in  the  top  right  hand  comer  of  the  Task-to- 
Competency  Rating  Form. 

2.  Read  the  tasks  on  the  Critical  AI/Robotics  Tasks  sheet.  Note  that  the  tasks  within  each  group  are 
similar,  except  for  the  last  group.  The  tasks  are  grouped  to  make  your  task  a  little  easier. 

3.  Read  the  list  of  competencies  carefully. 


4.  Read  the  description  for  Task  1 . 

5.  Read  Competency  1 .  Decide  whether  Competency  1  is  either  not  needed,  helpful,  or  essential  to 
performing  the  task  properly.  Write  the  number  that  corresponds  to  your  decision  in  the  square 
where  Competency  1  and  Task  1  intersect  according  to  the  scale  below: 

0 

not  needed 

This  competency  is  not  needed  to  perform  the  task.  Having  this 
competency  would  make  no  difference  in  the  performance  of  the  task. 

1 

helpful 

This  competency  is  helpful  in  performing  the  task,  but  not  essential.  The 
task  could  be  performed  without  this  competency,  but  the  task  would  be 
more  difficult  or  time-consuming. 

2 

essential 

This  competency  is  essential  to  the  performance  of  the  task.  Without  this 
competency,  you  would  not  be  able  to  perform  the  task. 

9 

don't  know 

If  you  are  unfamiliar  with  either  the  task  or  the  competency,  then  write  a 
question  mark  in  the  square. 

6.  Read  Competency  2  and  repeat  Step  5  above.  Write  the  number  corresponding  to  your  rating  in  the 
square  Competency  2  and  Task  1  intersect.  Repeat  this  step  for  each  Competency. 

7.  Repeat  Steps  5  and  6  for  each  task. 
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Critical  AI/Robotics  Tasks 


1 .  Seek  and  maintain  liaisons  with  academia/industry/govemment  to  keep  abreast  of  current 
and  emerging  AI/Robotics  practices  and  technologies. 

2.  Attend  courses,  symposia,  seminars,  and  conferences  to  keep  abreast  of  current  and 
emerging  AI/Robotics  practices  and  technologies. 

3.  Read  research  articles,  reports,  and  others  sources  (e.g.,  internet  messages)  to  keep  abreast  of 
current  and  emerging  AI/Robotics  practices  and  technologies. 

4.  Perform  overall  project  management  (e.g.,  monitor  budgets  and  timelines)  of  funded 
AI/Robotics  efforts. 

5.  Manage  daily  functions  of  funded  Al/robotics  efforts. 

6.  Oversee  contractors  performing  Al/robotics  work. 

7.  Evaluate  proposals  and  plans  for  Al/robotics  projects  and  programs. 

8.  Select  available  knowledge  sources  (e.g.,  subject  matter  experts,  data). 

9.  Perform  knowledge  engineering  (e.g.,  interviewing  subject  matter  experts,  evaluating  data). 

10.  Identify,  evaluate,  and  recommend  procurement  of  specific  hardware  and  software  tools. 

11.  Develop  Al/robotics  system  specifications  (e.g.,  for  environment,  software,  hardware,  and 
interface). 

12.  Evaluate  Al/robotics  software,  hardware,  and  tools. 

13.  Identify  and  assess  opportunities  to  apply  Al/robotics  to  improve  processes  and  systems. 

14.  Assist  users  in  refining  Al/robotics  system  requirements. 

15.  Build  systems  (e.g.,  computer  programs)  using  Al/robotics  principles  and  techniques. 

16.  Integrate  Al/robotics  components  into  systems. 

17.  Test  and  evaluate  Al/robotics  components  and/or  systems. 
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Standards 

for  the  Certification  of 
Civilians  in  the  U.S.  Army  in 
Artificial  Intelligence 


June  12,  1997 
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Certification  Performance  Standard 


Certificants  are  not  expected  to  be  experts  but  must  be  above  entry  level  in 
AI/Robotics.  They  must  demonstrate  competence  in  the  important  knowledge 
areas  in  Al/Robotics.  A  certificant  who  works  in  one  area  of  AI/Robotics  will  be 
able  to  work  in  another  area  of  AI/Robotics  with  minimal  additional  formal  or  on- 
the-job  AI/Robotics  training.  Therefore,  a  manager  can  be  confident  that  any 
AI/Robotics  certificant  hired  is  likely  to  be  able  to  successfully  perform  the 
AI/Robotics  aspects  of  the  job. 

Competence  in  core  and  supplemental  knowledges  must  be  demonstrated 
through  a  combination  of  the  certificant’s  training,  education,  and  experience. 
Strength  in  one  area  (e.g.,  experience)  may  compensate  for  weakness  in  another 
(e.g.,  education).  The  typical  certificant  will  have  an  advanced  degree  in  an 
AI/Robotics-related  field  and  at  least  6  months  of  relevant  job  experience.  Thesis 
research  in  an  area  of  AI/Robotics  will  count  for  the  necessary  experience; 
however,  advanced  degree  holders  who  did  not  complete  thesis  work  in 
AI/Robotics  must  demonstrate  the  equivalent  application  or  experience. 
Certificants  without  advanced  degrees  can  certify  in  the  competency  areas  through 
a  combination  of  training/education  and  hands-on  experience  in  AI/Robotics. 


Note:  All  core  competencies  can  be  satisfied  through  combinations  of  education 
or  experience  with  the  exception  of  Competency  5  (Methodologies  for  integrating 
AI  with  modeling,  optimization,  mathematical,  statistical,  or  other  computer 
programs).  This  competency  can  be  satisfied  only  through  experience  and  not 
through  education. 
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Core  Competencies  (each  certificant  must  fulfill  all  of  these) 

1 .  Expert/Knowledge-Based  Systems 

2.  Object-Oriented/Symbolic/Logic  Programming 

3.  Knowledge  Representation 

4.  Basic  Principles  of  AI 

5.  Methodologies  for  Integrating  AI  with  Modeling,  Optimization,  Mathematical,  Statistical, 
or  Other  Computer  Programs 

6.  Knowledge  Engineering 

Supplemental  Competencies  (each  certificant  must  fulfill  any  three  of  these) 

7.  Search  Strategies 

8.  Heuristic  Algorithms 

9.  Fuzzy  Logic 

10.  Neural  Networks 

1 1 .  Application  of  AI  Tools  (e.g.,  preprocessors,  shells) 

12.  Hybrid  Systems 

13.  Path  Planning 

14.  Blackboard  Systems 

15.  Natural  Language  Processing 

16.  Genetic  Algorithms 

17.  Case-Based  Reasoning 

1 8 .  Machine  Learning 

19.  Speech  Recognition/Synthesis 

20.  High-Level  Computer  Programming  Languages 

21.  Conceptual  Modeling  of  Problems  and  Solutions 

22.  Sensors 

23.  Sensor  Fusion 

24.  Basic  Principles  of  Robotics 

25.  Image  Processing  /  Computer  Vision 

26.  Control  Theory 

27.  Robot  Navigation 
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Descriptions  of  the  Core  Competencies 


1  Expert/Knowledge-Based  Systems 

This  competency  covers  the  control  architectures,  data  structures,  and  paradigms  used  in 
designing  and  building  knowledge-based  systems.  It  includes  production  systems,  object-oriented 
programming,  frames  and  default  reasoning,  logic-based  approaches  to  knowledge 
representation,  uncertainty  formalisms,  truth  maintenance,  and  expert  system  software. 

2  Object-Oriented/Symbolic/Logic  Programming 

This  competency  covers  the  fundamental  principles  of  programming  for  Artificial  Intelligence.  It 
focuses  on  symbolic,  logic,  or  object-oriented  programming,  and  upon  the  application  of 
development  language  and  common  AI  techniques  such  as  search  or  knowledge  representation.  It 
includes  principles  of  analysis,  design,  development,  and  programming. 

3  Knowledge  Representation 

This  competency  covers  the  control  architectures,  data  structures,  and  paradigms  used  in 
designing  and  building  AI  systems,  including  production  systems,  object  oriented  programming, 
frames  and  default  reasoning,  logic-based  approaches  to  knowledge  representation,  uncertainty 
formalisms,  truth  maintenance,  and  expert  system  software  tools. 

4  Basic  Principles  of  AI 

This  competency  covers  the  fundamentals  of  the  core  concepts  of  AI.  Its  primary  focus  is  the 
integration  between  AI  and  other  analytic  problem  solving  techniques.  It  includes  the  history  of 
AI;  the  applications  of  AI;  and  the  fundamentals  of  knowledge  representation,  expert  systems, 
and  AI  programming. 

5  Methodologies  for  Integrating  AI  with  Modeling,  Optimization,  Mathematical, 
Statistical,  or  Other  Computer  Programs 

This  competency  covers  hands-on  practical  experience  in  the  application  of  AI  techniques  or 
research  to  problem  solving.  It  involves  defining  the  scope  of  a  problem,  selecting  one  or  more 
appropriate  AI  techniques,  and  implementing  the  techniques  to  solve  a  problem. 

6  Knowledge  Engineering 

The  primary  focus  of  this  competency  is  on  knowledge  acquisition.  This  competency  covers  the 
acquisition  of  knowledge  from  a  variety  of  sources,  such  as  human  experts  and  written 
documentation. 
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Descriptions  of  the  Supplemental  Competencies 


7  Search  Strategies 

This  competency  addresses  a  fundamental  area  of  problem  solving:  the  state-space  search.  It 
covers  traditional  computational  search  strategies  as  well  as  Al-specific  strategies  such  as  depth- 
first,  breadth-first.  A*,  and  heuristic  search. 

8  Heuristic  Algorithms 

This  competency  covers  the  use  of  heuristics  in  problem  solving.  Heuristics  are  used  when  a 
problem’s  complexity  or  inherent  ambiguities  preclude  exact  (closed-form)  solutions.  It  covers 
the  discovery  of  “rules  of  thumb”  and  their  application  to  problem  solving. 

9  Fuzzy  Logic 

This  competency  covers  methods  of  modeling  uncertainty  utilizing  fuzzy  logic/set  theory.  It  may 
include  other  non-traditional  logic  constructs. 

10  Neural  Networks 

This  competency  covers  concepts  in  neural  networks  and  connectionist  models.  It  includes 
parallel  distributed  processing,  learning  algorithms,  and  applications.  Specific  networks  include 
Hopfield  networks,  bi-directional  associative  memories,  perceptrons,  feed-forward  networks  with 
back-propagation,  and  competitive  learning  networks  (including  self-organizing  and  Grossberg 
networks). 

11  Application  of  AI  Tools  (e.g.,  preprocessors,  shells) 

This  competency  covers  the  control  architectures,  data  structures,  and  paradigms  used  in 
designing  and  building  knowledge-based  systems.  It  includes,  but  is  not  limited  to,  production 
systems,  object-oriented  programming,  frames  and  default  reasoning,  logic-based  approaches  to 
knowledge  representation,  uncertainty  formalisms,  truth  maintenance,  and  expert  system 
software  tools.  The  candidate  must  demonstrate  the  application  of  a  commercial  AI  tool  or  shell. 
This  application  can  include  the  development  of  an  expert  system  or  other  shell  in  an  AI 
programming  language  ( see  competency  2). 

12  Hybrid  Systems 

This  competency  covers  the  method  of  integrating  two  or  more  AI  problem-solving  techniques  to 
solve  a  given  problem.  It  includes,  but  is  not  limited  to,  the  integration  of  an  AI  technique  with  a 
commercial  AI  shell. 
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13  Path  Planning 

This  competency  covers  the  fundamentals  of  state  space  search  for  the  purpose  of  route  selection 
or  path  planning.  It  can  be  used  to  simulate  intelligent  behavior  in  automated  decision-making  or 
used  in  robotic  applications.  Planning  topics  include  obstacle  avoidance,  task  planning,  and 
navigation.  Robotic  path  planning  emphasizes  vision  (including  binary  image  processing),  robot 
vision,  and  knowledge-based  vision  systems. 

14  Blackboard  Systems 

This  competency  covers  a  general  problem-solving  approach  that  requires  the  coordination  of 
multiple  processes  or  knowledge  sources  using  a  central  global  database  for  the  communication 
of  independent  asynchronous  knowledge  sources  or  processes.  Technical  implementation 
includes,  but  is  not  limited  to,  object-oriented  databases  and  object  request  brokers. 

15  Natural  Language  Processing 

This  competency  covers  the  concepts  and  methods  for  processing  natural  language  by  computer. 
It  includes  pattern  matching,  parsing,  the  role  of  the  dictionary  and  lexical  acquisition,  semantic 
interpretation,  anaphoric  reference,  plan  recognition,  discourse  analysis,  and  text  generation. 
Applications  include  natural  language  interfaces,  text  processing  systems,  advisory  systems,  and 
interaction  with  speech  recognizers. 

16  Genetic  Algorithms 

This  competency  covers  genetic  algorithms  and  other  evolutionary  based  AI  algorithms.  It 
includes,  but  is  not  limited  to,  artificial  life. 

17  Case-Based  Reasoning 

This  competency  covers  case-based  reasoning  as  a  type  of  analogical  reasoning  and  as  an 
alternative  method  for  building  systems.  It  includes  case  representation,  indexing,  retrieval, 
adaptation,  and  interpretive  case-based  reasoning.  It  also  covers  the  cognitive  model  that 
underlies  case-based  reasoning  and  the  model’s  implications  for  creativity,  decision  aids,  and 
education. 

18  Machine  Learning 

This  competency  covers  inductive  learning,  explanation-based  learning,  and  genetic  algorithms. 

It  includes  neural  networks,  decision  trees  and  graphs,  delayed-reinforcement  and  temporal- 
difference  learning,  and  computational  learning  theory.  It  focuses  on  the  underlying  concepts  of 
machine  learning  and  its  role  in  AI. 
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19  Speech  Recognition/Synthesis 

This  competency  emphasizes  processing  of  the  human  speech  waveform,  primarily  using  digital 
techniques.  It  covers  the  theory  of  speech  production  as  it  relates  to  signal  models  in  time  and 
frequency;  it  also  covers  the  measurement  of  model  parameters,  short-time  Fourier  spectrum,  and 
linear  predictor  coefficients.  It  may  include  speech  coding,  recognition  and  syntheses,  and 
speaker  recognition. 

20  High-Level  Computer  Programming  Languages 

This  competency  covers  traditional  computer  programming.  It  includes  analysis,  design, 
development,  and  programming.  The  focus  is  on  application  development. 

21  Conceptual  Modeling  of  Problems  and  Solutions 

The  focus  of  this  competency  is  on  problem  definition  and  structuring.  It  involves  the  application 
of  knowledge  representation  schemes  and  search  methods  to  describe  a  problem  space  prior  to 
the  selection  and  application  of  a  problem-solving  approach. 

22  Sensors 

This  competency  covers  the  fundamentals  of  sensing,  with  an  emphasis  on  vision.  It  includes 
binary  image  processing,  robot  vision,  and  knowledge-based  vision  systems. 

23  Sensor  Fusion 

This  competency  covers  the  combining  of  input  from  disparate  sensors  for  the  purpose  of 
decision-making.  It  includes,  but  is  not  limited  to,  the  application  of  measures  of 
uncertainty/belief.  In  robotics  applications,  it  includes  the  integration  of  sensor  inputs  to  establish 
situation  awareness. 

24  Basic  Principles  of  Robotics 

This  competency  covers  the  fundamentals  of  robot  technology  with  an  emphasis  on  programming 
concepts.  It  includes  robot  control,  robot  hardware,  the  mathematics  of  robot  control  (local  and 
global  coordinate  systems  and  transformations  between  them),  and  robot  programming 
languages.  Planning  includes  obstacle  avoidance,  task  planning,  and  navigation.  Sensing 
emphasizes  vision,  including  binary  image  processing,  robot  vision,  and  knowledge-based  vision 
systems. 
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25  Image  Processing  /  Computer  Vision 

This  competency  covers  the  basic  techniques  of  analysis  and  manipulation  of  pictorial  data  by 
computer.  It  includes  image  input/output  devices,  image  processing  software,  enhancement, 
segmentation,  property  measurement,  and  Fourier  analysis. 

26  Control  Theory 

This  competency  covers  modeling,  analysis,  and  the  fundamentals  of  design  for  feedback  control 
systems.  It  includes  state  equations,  transfer  function  representations,  stability,  performance 
measures,  control  systems  (hydraulic,  electrical,  mechanical,  and  pneumatic),  open  and  closed 
loops,  steady  state  and  transient  operation,  and  linear  and  non-linear  systems. 

27  Robot  Navigation 

This  competency  covers  the  fundamentals  of  robot  technology.  It  emphasizes  sensing  and 
planning.  Planning  topics  include  obstacle  avoidance,  task  planning,  and  navigation.  Sensing 
topics  include  vision  (including  binary  image  processing),  robot  vision,  and  knowledge-based 
systems. 
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Candidates  must  demonstrate  sufficient  mastery  in  all  six  core  competencies  and  three 
supplemental  competencies.  This  may  be  done  for  each  applicable  competency  area  via  related 
experience,  relevant  course  work,  or  combinations  of  both.  This  section  describes  how  a 
candidate  can  meet  the  requirements  for  a  competency.  The  typical  certificant  will  have  an 
advanced  degree  in  an  AI/Robotics-related  field  and  at  least  6  months  of  relevant  job  experience. 
Thesis  research  in  an  area  of  AI/Robotics  will  count  for  the  necessary  experience;  however, 
advanced  degree  holders  who  did  not  complete  thesis  work  in  AI/Robotics  must  demonstrate  the 
equivalent  application  or  experience.  Certificants  without  advanced  degrees  can  certify  in  the 
competency  areas  through  a  combination  of  training/education  and  hands-on  experience  in 
AI/Robotics. 

Course  Work 

A  list  of  courses  that  may  be  used  to  fulfill  the  certification  requirements  for  each  competency 
follows.  The  list  of  courses  is  not  meant  to  be  exhaustive.  If  a  candidate  lists  a  course  that  is  not 
on  this  list,  the  course  information  should  be  compared  with  the  courses  that  are  on  the  list  to 
help  determine  whether  the  course  meets  the  requirements  for  certification.  The  list  of  courses 
includes  a  prototype  course  for  each  competency.  A  prototype  course  for  a  competency  is  the 
course  that  best  matches  the  requirements  for  that  competency.  A  Prototype  Courses  Linked  to 
Competencies  table  is  also  provided  to  graphically  provide  some  of  the  information  contained  in 
the  list  of  courses. 

For  Competency  5  (Methodologies  for  integrating  AI  with  modeling,  optimization,  mathematical, 
statistical,  or  other  computer  programs),  courses  cannot  be  used  for  either  full  or  partial  credit; 
this  competency  can  be  satisfied  only  through  experience. 

The  candidate  must  demonstrate  the  application  of  AI  tools  to  solve  a  problem.  The  application 
must  be  beyond  the  scope  of  a  mere  course  project.  Al-related  thesis  research  or  six  months  of 
applied  experience  is  the  minimum  requirement. 

A  course  may  partially  or  completely  fulfill  the  requirements  for  a  competency.  A  course  that 
partially  satisfies  a  competency  may  be  combined  with  experience  to  fulfill  the  requirements  for 
a  competency  (as  described  below),  but  it  cannot  be  combined  with  other  courses. 

Experience 

The  certification  review  panel  will  look  for  active  involvement  in  the  design  or  development  of 
AI  systems  or  active  involvement  in  the  verification,  validation,  testing,  or  evaluation  of  AI 
systems.  The  experiences  must  equip  a  person  with  sufficient  knowledge  to  make  appropriate 
choices  when  solving  a  problem.  Six  months  of  relevant  experience,  where  at  least  50%  of  the 
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candidate’s  time  was  devoted  to  relevant  AI/Robotics  activities,  will  typically  be  required  to 
receive  full  credit  for  a  competency. 

Experience  +  Course  Work  Combinations 

The  Board  will  look  for  active  involvement  in  AI  projects  in  conjunction  with  related  course 
work.  The  completion  of  a  short  course  combined  with  active  involvement  in  parts  of  related 
projects  or  technical  oversight  (of  design,  development,  or  testing)  may  be  used  to  fulfill  the 
requirements  for  a  competency.  Typically,  courses  must  be  at  least  two  days  in  duration  (or  2 
credit  hours  for  courses  that  last  a  semester)  and  relevant  experience  must  be  at  least  3  months, 
where  at  least  50%  of  the  candidate’s  time  was  devoted  to  relevant  AI/Robotics  activities,  to 
receive  full  credit  for  a  competency. 

Kinds  of  Experiences  or  Combinations  that  are  Enough  for  most  AI/Robotics  Competencies 

•  participated  in  a  co-op  program 

•  was  a  visiting  scholar 

•  performed  job-related  or  graduate  research 

•  was  actively  involved  in  work  on  a  relevant  component  of  a  project  and  took  a  short 
course 

•  provided  technical  oversight  of  design,  development,  or  testing  to  a  project/program  and 
took  a  short  course 

•  active  participation  in  selection  of  techniques 

•  active  design  of  solutions  and  approaches 

Kinds  of  Experiences  or  Combinations  that  are  Not  Enough  for  most  AI/Robotics 
Competencies 

•  oversight  of  projects  (i.e.,  program  management) 

•  attended  a  one-day  class  or  workshop 

•  attended  a  conference  tutorial  (While  we  encourage  professionals  to  participate  in 
conference  tutorials  in  order  to  remain  current,  they  do  not  satisfy  the  requirements  for 
certification.) 

•  membership  in  an  AI/Robotics-related  professional  organization 

•  developed  solutions  using  traditional  computational/mathematical/optimization 
techniques  (especially  using  a  single  COTS  shell) 

Detailed  Examples 

The  examples  above,  of  kinds  of  experiences  and  combinations  of  experience  and  course  work 
that  are  either  sufficient  or  insufficient  for  certification,  are  general  in  nature.  More  detailed 
examples,  for  each  core  competency,  are  shown  later. 
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Full  Credit  P  =  Partial  Credit 


__________ _ 12 

Prototype-Course  to  Competency  Mapping 


ID  Prototype-Course  Title 

1 1  Robotics 

Additional  Courses: 

1 1  Robotics 

29  Introduction  to  Robotics 
189  Robotics 

246  Advanced  Topics  in  AI:  Robotics 
297  Robot  Navigation 


Description  of  Prototype  Course: 

The  course  introduces  the  fundamentals  of  robot  technology 
with  an  emphasis  on  programming  concepts.  Topics  covered 
in  robot  control  are  robot  hardware,  the  mathematics  of 
robot  control  (local  and  global  coordinate  systems  and 
transformations  between  them),  and  robot  programming 
languages.  Planning  topics  include  obstacle  avoidance,  task 
planning,  and  navigation.  Sensing  emphasizes  vision, 
including  binary  image  processing,  robot  vision,  and 
knowledge-based  vision  systems.  Programming  assignments 
and  a  laboratory  are  included. 


Competency 

Number  Competency  Title 

22 

Sensors 

23 

Sensor  Fusion 

24 

Basic  Principles  of  Robotics 

26 

Control  Theoiy 

27 

Robot  Navigation 

13 

Path  Planning 

ID  Prototype-Course  Title 

46  Expert  Systems 

Additional  Courses: 

15  Knowledge-based  Systems  and  AI 

45  Introduction  to  Experts  Systems 

46  Expert  Systems 

83  Expert  Systems  in  Ind.  Engineering 
1 18  Expert  Systems:  Practical  Applications  of  AI 
132  Knowledge-based  Systems 
151  Expert  Systems 
181  Knowledgebases 

243  Advanced  Topics  in  AI:  Expert  Systems 


Description  of  Prototype  Course: 

This  course  explores  the  control  architectures,  data 
structures,  and  paradigms  used  in  designing  and  building 
knowledge-based  systems.  Topics  include  production 
systems,  object-oriented  programming,  frames  and  default 
reasoning,  logic-based  approaches  to  knowledge 
representation,  uncertainty  formalisms,  truth  maintenance, 
and  expert  system  software  tools.  A  number  of  case  studies 
from  the  expert  systems  literature  are  also  considered.  The 
class  has  a  laboratory  in  which  expert  system  shells  are  used 
to  develop  concepts  discussed  in  the  lectures. 

Competency 

Number  Competency  Title 

I  Expert/Knowledge-Based  Systems 

3  Knowledge  Representation 

6  Knowledge  Engineering 

2 1  Conceptual  Modeling  of  Problems  and 

Solutions 

I I  Application  of  AI  Tools  (e.g.  preprocessors, 
shells) 
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ID  Prototype-Course  Title 

55  Fuzzy  Logic 

Additional  Courses: 

55  Logics  for  Computer  Science 


ID  Prototype-Course  Title 

67  Neural  Networks 

Additional  Courses: 

67  Neural  Networks 
1 28  Neural  Network  Short  Course 

135  Neural  Computing 
1 4 1  Neural  Networks 

160  Applied  Neural  Network  computing 

161  Artificial  Neural  Systems  and  Neural  Computing 

162  Applying  Neural  Computing 

163  Advanced  Neural  Computing 

164  Empirical  Modeling  Using  Neural  Networks 
204  Neural  net  and  Parallel  Systems 

26 1  Applying  Neural  Networks 


Description  of  Prototype  Course: 

This  course  introduces  the  student  to  formal  systems,  their 
syntax,  semantics,  manipulation,  and  interpretation. 
Propositional  and  predicate  logics  are  introduced  as 
mathematical  systems  and  as  formal  models  of  (parts  of) 
human  reasoning.  Applications  emphasize  the  logical 
investigation  of  the  properties  of  various  data  structures. 
Several  alternative  logics  are  presented,  such  as  modal, 
multi-valued,  fuzzy,  temporal,  intuitionistic,  and  inductive. 
These  are  discussed  as  potential  tools  in  knowledge 
representation  and  cognitive  science. 

Competency 

Number  Competency  Title 

9  Fuzzy  Logic 


Description  of  Prototype  Course: 

This  course  provides  an  introduction  to  concepts  in  neural 
networks  and  connectionist  models.  Topics  include  parallel 
distributed  processing,  learning  algorithms  and  applications. 
Specific  networks  discussed  include  Hopfield  networks, 
bidirectional  associative  memories,  perceptrons,  feedforward 
networks  with  back  propagation,  and  competitive  learning 
networks,  including  self-organizing  and  Grossberg  networks 

Competency 

Number  Competency  Title 

10  Neural  Networks 
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ID  Prototype- Course  Title 

87  Building  Expert  Systems 

Additional  Courses: 

87  Building  Expert  Systems 


ID  Prototype-Course  Title 

1 16  Programming  in  Prolog 

Additional  Courses: 

1 1 6  Programming  in  Prolog 

244  Advanced  Topics  in  Al:  Logic  Programming 


Description  of  Prototype  Course: 

Building  Expert  Systems  is  an  introductory 
through-intermediate  level  guide  to  knowledge  engineering 
and  expert  system  development.  Students  are  guided  via 
lecture  and  computer  lab  through  the  construction  of  a  small 
but  fully  functioning  expert  system.  The  course  contains  an 
historical  and  theoretical  overview  of  the  discipline  of 
Artificial  Intelligence,  a  comprehensive  review  of  successful 
expert  systems  (e.g.,  corporate,  academic,  and  military 
implementations),  and  reviews  those  successful  strategies 
used  in  AI  development.  The  primary  focus  of  this  course  is 
on  knowledge  elicitation  and  representation.  The  purpose  of 
the  course  is  to  provide  specialized  training  in  the 
development  of  Expert/Knowledge-based  Systems  to  Army 
Officers  in  the  Functional  Areas  49  and  53  and  DA  Civilians 
working  with  AI. 

Competency 

Number  Competency  Title 

I  Expert/Knowledge-Based  Systems 

3  Knowledge  Representation 

4  Basic  Principles  of  AI 

6  Knowledge  Engineering 

2 1  Conceptual  Modeling  of  Problems  and 

Solutions 

I I  Application  of  AI  Tools  (e.g.  preprocessors, 
shells) 


Description  of  Prototype  Course: 

This  course  provides  an  introduction  to  Prolog,  the 
foundations  of  logic  programming,  and  current  research  on 
applications  of  logic  within  artificial  intelligence.  The  course 
proceeds  on  two  parallel  tracks,  with  each  class  session 
divided  about  equally  between  the  two.  One  track  focuses  on 
Prolog  programming.  It  begins  with  a  thorough  treatment  of 
Prolog,  and  then  considers  applications  of  Prolog  to  natural 
language  processing  and  other  areas.  The  other  track  focuses 
on  theoretical  issues.  It  begins  with  a  review  of  firstorder 
logic  and  the  resolution  principle  and  then  considers 
theoretical  aspects  of  logic  programming,  including  the 
semantics  of  logic  programs  and  alternative  proof 
procedures.  Finally,  it  examines  some  of  the  alternatives  to 
first-order  logic  that  Al  researchers  have  considered  recently 
to  overcome  inadequacies  of  traditional  logical  method)  for 
building  intelligent  systems.  These  alternatives  include  modal 
logics  for  representing  and  reasoning  about  knowledge  and 
belief,  and  nonmonotonic  and  default  logics.  Assignments 
include  problem  sets  and  a  number  of  Prolog  programs. 

Competency 

Number  Competency  Title 

2  Object-oriented/Symbolic/Logic  Programming 

20  High-Level  Computer  Programming  Languages 
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15 


ID  Prototype-Course  Title 

119  AI  Research/Experience 


ID  Prototype-Course  Title 

134  Genetic  Algorithms 

Additional  Courses: 

134  Genetic  Algorithms 


ID  Prototype-Course  Title 

137  Object-oriented  Programming 

Additional  Courses: 

137  Object-oriented  Programming 
157  Object  Oriented  Design  and  Analysis 
1 79  C++  Object  Oriented  Programming 
203  Object  Oriented  Programming 
21 1  C++  Programming 


Description  of  Prototype  Course: 

Application  development  experience,  thesis  research  or 

Competency 

Number  Competency  Title 

5  Methodologies  for  Integrating  AI 

with  Modeling,  Optimization, 
Mathematical,  Statistical,  or  Other 
Computer  Programs 

21  Conceptual  Modeling  of  Problems  and 

Solutions 

1 1  Application  of  AI  Tools  (e.g.  preprocessors, 

shells) 


Description  of  Prototype  Course: 

This  course  will  discuss  Genetic  Algorithms  and  other 
evolutionary  based  AI  algorithms,  including  a  small 
component  on  Artificial  Life.  GAs  are  easy  to  understand, 
easy  to  program,  and  very  powerful  tools.  They  are  being 
used  to  do  all  kinds  of  things  these  days,  including  machine 
learning,  gas  pipeline  routing,  airplane  wing  design,  function 
optimization,  scheduling  (including  scheduling  task/processor 
assignments  in  parallel  systems),  resource  allocation,  and 
other  types  of  hard  problems  for  which  it  is  not  possible  to 
directly  compute  the  best  answer. 

Competency 

Number  Competency  Title 

16  Genetic  Algorithms 


Description  of  Prototype  Course: 

Principles  of  object-oriented  analysis  and  design, 
development,  and  programming  Includes  relation  Dip. 

Between  object-oriented  design  concepts  and  software 
engineering  principle,  techniques  of  object-oriented  design 
and  programming  and  appoint  object-oriented  technique. 

Competency 

Number  Competency  Title 

2  Object-oriented/Symbolic/Logic  Programming 

20  High-Level  Computer  Programming  Languages 
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ID  Prototype-Course  Title 

1 50  LISP  Programming 

Additional  Courses: 

79  Porting  LISP  Applications  to  Stock  Hardware 

80  Intro  to  LISP  Programming 
150  LISP  Programming 

1 52  Introduction  to  Common  LISP 


ID  Prototype-Course  Title 

167  Control  theory 

Additional  Courses: 

1 4  Theory  and  Applications  of  Fuzzy  Control 

33  Advanced  Systems  Control 

34  Automatic  Controls 

158  Adaptive  Control 

159  Hybrid  Controls 
167  Control  theory 
221  Feedback  Controls 


Description  of  Prototype  Course: 

Introduction  to  computer  programming  in  Common  LISP, 
with  a  focus  on  preparing  the  student  for  using  Common 
LISP  and  AI  programming  techniques  to  solve  problems  in 

Competency 

Number  Competency  Title 

2  Object-oriented/Symbolic/Logic  Programming 

20  High-Level  Computer  Programming  Languages 


Description  of  Prototype  Course: 

Modeling,  analysis  and  an  introduction  to  design  for  feedback 
control  systems.  Topics  include  state  equation,  and  transfer 
function  representations,  stability,  performance  measures, 
hydraulic,  electrical,  mechanical  and  pneumatic  control 
systems,  open  and  closed  loops,  steady  state  and  transient 
operation  and  linear  and  non-linear  systems. 

Competency 

Number  Competency  Title 

26  Control  Theory 


Certification  Standards  for  Artificial  Intelligence 


E-  17 


June  12, 1997 


17 


ED  Prototype-Course  Title 

212  ITOR 

Additional  Courses: 

212  ITOR 


Description  of  Prototype  Course: 

The  ITOR  course  provides  the  student  with  an  introduction 
to  the  world  of  Artificial  Intelligence  (AI).  The  fields  in  AI 
that  the  student  explores  includes  Neural  Networks,  Expert 
Systems  and  Object-Oriented  Programming.  The  key  focus 
of  the  course  is  to  provide  insight  into  the  integration 
between  this  new  knowledge  and  other  analytic  problems 
solving  techniques.  The  course  contains  a  touch  of  theory 
coupled  with  ample  hands-on  application.  This  combination 
provides  students  with  an  ability  to  solve  reasonable  size 
problems  in  each  field  before  graduation.  The  course  goals 
are  to: 

1.  Obtain  a  working  knowledge  AI  and  expert  systems 
concepts. 

2.  Identify  appropriate  applications  of  AI  Technology  to 
real  world  problems. 

3.  Create  AI  applications  using  neural  networks,  rule  based 
expert  systems  and  object-oriented  programming. 

4.  Experience  the  development  processes  involved  with 
EXSYS,  SNNAP,  and  Kappa  PC. 

5.  Create  a  working  prototype  of  a  project  from  the 

Competency 

Number  Competency  Title 

1  Expert/Knowledge-Based  Systems 

2  Object-oriented/Symbolic/Logic  Programming 

3  Knowledge  Representation 

4  Basic  Principles  of  AI 

21  Conceptual  Modeling  of  Problems  and 

Solutions 

10  Neural  Networks 

1 1  Application  of  AI  Tools  (e.g.  preprocessors, 

shells) 

12  Hybrid  Systems 
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ID  Prototype-Course  Title 

2 1 6  Artificial  Intelligence 

Additional  Courses: 

20  Introduction  to  AI 
35  Introduction  to  AI 

89  Advanced  AI 
109  Advanced  AI 
1 1 7  Problem-solving  Methods  in  AI 

1 3 1  Introduction  to  AI 
216  Artificial  Intelligence 


Description  of  Prototype  Course: 

The  course  addresses  two  major  areas:  representation  and 
reasoning.  Topics  focusing  on  representation  include 
symbolic,  production  rule,  frame,  object,  and  semantic  net 
representational  schemes.  Topics  focusing  on  reasoning  may 
include  weak  methods  (path  finding,  constraint  propagation, 
games,  general  heuristic  search,  decomposition,  general 
problem  solver,  generate  and  test,  hill  climbing),  deduction, 
explanation,  uncertainty,  non-monotonic  reasoning, 
temporal  reasoning,  model-based  reasoning,  and  planning. 
Applications  may  include  production  rule  systems,  natural 
language  examples,  learning,  and  planning.  Common  LISP  is 
introduced  as  a  tool  for  expressing  symbolic  knowledge. 


Competency 

Number  Competency  Title 

1  Expert/Knowledge-Based  Systems 

2  Object-oriented/Symbolic/Logic  Programming 

3  Knowledge  Representation 

4  Basic  Principles  of  AI 

6  Knowledge  Engineering 

7  Search  Strategies 

20  High-Level  Computer  Programming  Languages 

21  Conceptual  Modeling  of  Problems  and 
Solutions 

8  Heuristic  Algorithms 

15  Natural  Language  Processing 


ID  Prototype-Course  Title 

232  Speech  Processing 

Additional  Courses: 

232  Speech  Processing 


Description  of  Prototype  Course: 

The  course  emphasizes  processing  of  the  human  speech 
waveform,  primarily  using  digital  techniques.  Theory  of 
speech  production  as  related  to  signal  models  in  time  and 
frequency  domains  is  covered,  as  well  as  the  measurement  of 
model  parameters,  short-time  Fourier  spectrum,  and  linear 
predictor  coefficients.  Speech  coding,  recognition  and 
synthesis,  and  speaker  recognition  are  also  included.  Some 
consideration  is  given  to  speech  processing  hardware. 
Software  projects  are  required. 


Competency 

Number  Competency  Title 

19  Speech  Recognition/Synthesis 
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ID  Prototype-Course  Title 

235  Natural  Language  Processing 

Additional  Courses: 

235  Natural  Language  Processing 


ID  Prototype-Course  Title 

237  Image  Processing 

Additional  Courses: 

142  Statistical  Pattern  Recognition 
195  Image  Processing/Robot  Vision 
237  Image  Processing 


ID  Prototype-Course  Title 

238  Computer  Vision 

Additional  Courses: 

1 2  Computer  Vision  and  Image  Processing 
170  Machine  vision 

174  Statistical  Techniques  in  computer  Vision 
238  Computer  Vision 


Description  of  Prototype  Course: 

Natural  Language  Processing  This  covers  the  concepts  and 
methods  for  processing  natural  language  by  computer. 

Topics  include  pattern  matching,  parsing,  the  role  of  the 
dictionary  and  lexical  acquisition,  semantic  interpretation, 
anaphoric  reference,  plan  recognition,  discourse  analysis,  and 
text  generation.  Applications  are  drawn  from  natural 
language  interfaces,  text  processing  systems,  advisory 
systems,  and  interaction  with  speech  recognizers.  A  "hands 
on"  natural  language  processing  development  tool  is  used. 

Competency 

Number  Competency  Title 

15  Natural  Language  Processing 


Description  of  Prototype  Course: 

An  introduction  to  basic  techniques  of  analysis  and 
manipulation  of  pictorial  data  by  computer.  Image 
input/output  devices,  image  processing  software, 
enhancement,  segmentation,  property  measurement,  Fourier 
analysis.  Computer  encoding,  processing,  and  analysis  of 

Competency 

Number  Competency  Title 

25  Image  Processing  /  Computer  Vision 


Description  of  Prototype  Course: 

Image  processing:  edge  detection,  segmentation,  local 
features,  shape  and  region  description  in  2D  and  3D.  Insights 
from  human  vision.  Representation  for  vision:  object 
models,  synthetic  images,  matching,  gaps,  algorithms. 
Inference,  production  system,  syntactic  networks.  Planning, 
spatial  reasoning  for  robot  vision. 

Competency 

Number  Competency  Title 

25  Image  Processing  /  Computer  Vision 
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ID  Prototype-Course  Title 

242  Artificial  Intelligence  Programming 

Additional  Courses: 

1  Symbolic  and  Logic  Programming 
242  Artificial  Intelligence  Programming 


ID  Prototype-Course  Title 

249  Logic,  Language  and  Machines 

Additional  Courses: 

249  Logic,  Language  and  Machines 


Certification  Standards  for  Artificial  Intelligence 


Description  of  Prototype  Course: 

This  course  focuses  on  implementing  AI  techniques  in 
Common  LISP.  As  such,  it  has  two  main  themes:  the 
algorithms  needed  to  implement  A I  techniques  efficiently, 
and  the  LISP  techniques  needed  to  put  the  algorithms  into 
practice.  The  course  consists  primarily  of  programming 
projects.  AI  Programming  techniques  may  include  a  deeper 
look  at  heuristic  search  and  efficiency  issues  of  search 
programs,  chart  parsing  for  efficiently  parsing  natural 
language,  object-oriented  and  frame-based  representations, 
rule-based  expert  systems,  etc.  LISP  issues  include 
programming  methods  as  well  as  LISP  functions  and  data 
structures.  Programming  methods  may  include  symbolic 
programming,  using  lexical  closures,  memorization,  and  a 
review  of  recursive  techniques.  LISP  functions  and  data 
structures  may  include  the  Common  LISP  Object  System 
(CLOS),  hash  tables,  functions  as  first-class  objects,  macros, 
structures,  and  lists. 

Competency 

Number  Competency  Title 

2  Object-oriented/Symbolic/Logic  Programming 

20  High-Level  Computer  Programming  Languages 


Description  of  Prototype  Course: 

The  overall  objective  of  this  course  is  to  introduce  the 
student  to  formal  systems:  their  syntax,  semantics, 
manipulation,  and  interpretation.  Computer  programs  are 
themselves  formal  systems,  and  the  subject  matter  underlies 
work  in  the  areas  of  correctness,  consistency,  and 
completeness  proofs  for  programs.  Topics  include  an 
introduction  to  symbolic  logic,  presented  as  an  attempt  to 
formally  model  human  reasoning;  and  inductive  logic  in  the 
context  of  modeling  the  reasoning  involved  in  drawing 
conclusions  from  limited,  possibly  faulty  information  (this 
material  is  of  particular  interest  for  its  potential  in  artificial 
intelligence  research  and  development  in  the  area  of 
cognitive  modeling).  Finally,  a  survey  is  presented  of  other 
logical  systems  of  interest  in  the  areas  of  knowledge 
representation  and  reasoning  under  certainty:  topics 
discussed  will  include  modal,  fuzzy,  and  probabilistic  logics. 

Competency 

Number  Competency  Title 

2  Object-oriented/Symbolic/Logic  Programming 

9  Fuzzy  Logic 


June  12, 1997 


E  -  21 


21 


ID  Prototype-Course  Title 

296  Machine  Learning 

Additional  Courses: 

296  Machine  Learning 


Description  of  Prototype  Course: 

Survey  of  major  research  areas:  inductive  learning, 
explanation-based  learning,  and  genetic  algorithms.  Topics: 
neural  networks,  decision  trees  and  graphs, 
delayed-reinforcement  and  temporal-difference  learning,  and 
computational  learning  theory.  Focuses  on  the  underlying 
concepts  and  the  role  of  machine  learning  in  Al. 
Representative  systems  described.  Prerequisites:  221  or 
consent  of  instructor,  and  ability  to  write  computer 
programs  in  one  or  more  commonly  used  languages. 

Competency 

Number  Competency  Title 

16  Genetic  Algorithms 

17  Case -Based  Reasoning 

18  Machine  Learning 


ID  Prototype-Course  Title 

347  Case-Based  Reasoning 

Additional  Courses: 

298  Case-based  reasoning 


Description  of  Prototype  Course: 

Case-based  reasoning  is  a  kind  of  analogical  reasoning  and  an 
alternative  method  for  building  expert  systems.  Topics 
include  case  representation,  indexing,  and  retrieval, 
adaptation,  interpretive  case-based  reasoning,  the  cognitive 
model  case-based  reasoning  implies,  and  its  implications  for 
creativity,  decision  aiding,  and  education. 

Competency 

Number  Competency  Title 

17  Case-Based  Reasoning 
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Examples  of  Experience  or  the  Combination  of  Experience  +  Course  Work  for  Each  Core  Competency 
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Competency  Example  of  Enough  Experience  Example  of  Not  Enough  Experience 

5.  Methodologies  for  Derived  the  definition  and  solution  of  a  multi-  Developed  a  stand-alone  AI  system. 

integrating  AI  with  domain  problem;  participated  in  an 

modeling,  optimization,  interdisciplinary  team  to  provide  AI  support  to  a 

mathematical,  statistical,  project;  developed  an  AI  module  as  a  component 
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